panic on 2.6.30-rc6

panic on 2.6.30-rc6

[Burak Gorkemli]

Hi,

I am having kernel panic while doing video streaming tests with ccid3, using kernel version 2.6.30-rc6, pulled from Gerrit's
tree. The crash happens on the receiver node (there is a sender node and a receiver node, both Linux boxes running 2.6.30-rc6) and it is quite reproducible. The console output is as follows:

[<>] ? do_invalid_op
[<>] do_invalid_op
[<>] ? ccid3_hc_tx_no_feedback_timer
[<>] ? __wake_up
[<>] ? __kfree_skb
[<>] ? trace_hardirqs_off_thunk
[<>] error_code
[<>] ? __mod_timer
[<>] ? do_invalid_op
[<>] ? ccid3_hc_tx_no_feedback_timer
[<>] run_timer_softirq
[<>] ? ccid3_hc_tx_no_feedback_timer
[<>] ? ccid3_hc_tx_no_feedback_timer
[<>] __do_softirq
[<>] ? __do_softirq 
<IRQ> [<>] ? irq_exit
[<>] ? smp_apic_timer_interrupt
[<>] ? apic_timer_interrupt
[<>] ? native_safe_halt
[<>] ? default_idle
[<>] ? cpu_idle
[<>] ? rest_init
[<>] ? start_kernel
[<>] ? i386_start_kernel

Any comments?

Thanks.

 --
Burak

yet another panic on 2.6.30-rc6

[Burak Gorkemli]

The console output for this one is as follows:

[<>] do_trap
[<>] ? do_invalid_op
[<>] do_invalid_op
[<>] ? ccid3_hc_tx_no_feedback_timer
[<>] ? e1000_clean_rx_irq
[<>] ? trace_hardirqs_off_thunk
[<>] error_code
[<>] ? __mod_timer
[<>] ? do_invalid_op
[<>] ? ccid3_hc_tx_no_feedback_timer
[<>] run_timer_softirq
[<>] ? ccid3_hc_tx_no_feedback_timer
[<>] ? ccid3_hc_tx_no_feedback_timer
[<>] __do_softirq
[<>] ? __do_softirq
<IRQ> [<>] ? irq_exit
[<>] ? smp_apic_timer_interrupt
[<>] ? apic_timer_interrupt
[<>] ? native_safe_halt
[<>] ? default_idle
[<>] ? cpu_idle
[<>] ? rest_init
[<>] ? start_kernel
[<>] ? i386_start_kernel

 --
Burak



----- Original Message ----
> From: Burak Gorkemli <burakgmail-dccp@yahoo.com>
> To: dccp@vger.kernel.org
> Sent: Saturday, June 6, 2009 1:58:59 AM
> Subject: panic on 2.6.30-rc6
> 
> 
> Hi,
> 
> I am having kernel panic while doing video streaming tests with ccid3, using 
> kernel version 2.6.30-rc6, pulled from Gerrit's
> tree. The crash happens on the receiver node (there is a sender node and a 
> receiver node, both Linux boxes running 2.6.30-rc6) and it is quite 
> reproducible. The console output is as follows:
> 
> [<>] ? do_invalid_op
> [<>] do_invalid_op
> [<>] ? ccid3_hc_tx_no_feedback_timer
> [<>] ? __wake_up
> [<>] ? __kfree_skb
> [<>] ? trace_hardirqs_off_thunk
> [<>] error_code
> [<>] ? __mod_timer
> [<>] ? do_invalid_op
> [<>] ? ccid3_hc_tx_no_feedback_timer
> [<>] run_timer_softirq
> [<>] ? ccid3_hc_tx_no_feedback_timer
> [<>] ? ccid3_hc_tx_no_feedback_timer
> [<>] __do_softirq
> [<>] ? __do_softirq 
> [<>] ? irq_exit
> [<>] ? smp_apic_timer_interrupt
> [<>] ? apic_timer_interrupt
> [<>] ? native_safe_halt
> [<>] ? default_idle
> [<>] ? cpu_idle
> [<>] ? rest_init
> [<>] ? start_kernel
> [<>] ? i386_start_kernel
> 
> Any comments?
> 
> Thanks.
> 
> --
> Burak
> 

Re: yet another panic on 2.6.30-rc6

[Gerrit Renker]

Hi Burak,

thank you for raising this. We need to isolate the cause of the problem.

The dccp test tree is based on the netdev-2.6 tree maintained by Dave
Miller, which itself contains other trees (I think the total number of
kernel trees is ~130).

I am testing this tree myself, though mostly with audio streaming over
WiFi links, using the paraslash streamer.

From looking through the trace it looks as though this is a timer
problem. Any chance of getting the EIP value to see where it 

Regarding reproducability -- unfortunately I had just put in a new
revision onto eden-feed. Can you please consider pulling again from

	git://eden-feed.erg.abdn.ac.uk/dccp_exp [subtrees dccp/ccid4]

If this problem persists, it would be good to CC: netdev@vger as well.

It is quite possible that the problem may have cleared itself as a
result of using more recent sources. The dccp test tree gets a fresh
base at least every week, to avoid transient problems.

Thanks in advance
Gerrit


Quoting Burak Gorkemli:
|
| The console output for this one is as follows:
|
| [<>] do_trap
| [<>] ? do_invalid_op
| [<>] do_invalid_op
| [<>] ? ccid3_hc_tx_no_feedback_timer
| [<>] ? e1000_clean_rx_irq
| [<>] ? trace_hardirqs_off_thunk
| [<>] error_code
| [<>] ? __mod_timer
| [<>] ? do_invalid_op
| [<>] ? ccid3_hc_tx_no_feedback_timer
| [<>] run_timer_softirq
| [<>] ? ccid3_hc_tx_no_feedback_timer
| [<>] ? ccid3_hc_tx_no_feedback_timer
| [<>] __do_softirq
| [<>] ? __do_softirq
| <IRQ> [<>] ? irq_exit
| [<>] ? smp_apic_timer_interrupt
| [<>] ? apic_timer_interrupt
| [<>] ? native_safe_halt
| [<>] ? default_idle
| [<>] ? cpu_idle
| [<>] ? rest_init
| [<>] ? start_kernel
| [<>] ? i386_start_kernel
|
|  --
| Burak
|
|
|
| ----- Original Message ----
| > From: Burak Gorkemli 
| > To: dccp@vger.kernel.org
| > Sent: Saturday, June 6, 2009 1:58:59 AM
| > Subject: panic on 2.6.30-rc6
| >
| >
| > Hi,
| >
| > I am having kernel panic while doing video streaming tests with ccid3, using
| > kernel version 2.6.30-rc6, pulled from Gerrit's
| > tree. The crash happens on the receiver node (there is a sender node and a
| > receiver node, both Linux boxes running 2.6.30-rc6) and it is quite
| > reproducible. The console output is as follows:
| >
| > [<>] ? do_invalid_op
| > [<>] do_invalid_op
| > [<>] ? ccid3_hc_tx_no_feedback_timer
| > [<>] ? __wake_up
| > [<>] ? __kfree_skb
| > [<>] ? trace_hardirqs_off_thunk
| > [<>] error_code
| > [<>] ? __mod_timer
| > [<>] ? do_invalid_op
| > [<>] ? ccid3_hc_tx_no_feedback_timer
| > [<>] run_timer_softirq
| > [<>] ? ccid3_hc_tx_no_feedback_timer
| > [<>] ? ccid3_hc_tx_no_feedback_timer
| > [<>] __do_softirq
| > [<>] ? __do_softirq
| > [<>] ? irq_exit
| > [<>] ? smp_apic_timer_interrupt
| > [<>] ? apic_timer_interrupt
| > [<>] ? native_safe_halt
| > [<>] ? default_idle
| > [<>] ? cpu_idle
| > [<>] ? rest_init
| > [<>] ? start_kernel
| > [<>] ? i386_start_kernel
| >
| > Any comments?

Re: yet another panic on 2.6.30-rc6

[Burak Gorkemli]

Hi Gerrit,

Sorry for the late reply, I have just found the time today to pull the latest
tree from your site and test it. It still crashes unfortunately, giving similar
console output, as given below. As for the transient problems, I had the
same issue with an older kernel, 2.6.26-rc5, again pulled from your site.

BTW, how to print EIP after the panic?

The console output of the latest panic is as follows:

[<>] ? ccid3_hc_tx_no_feedback_timer
[<>] ? default_spin_lock_flags
[<>] ? elv_queue_empty
[<>] ? blk_run_queue
[<>] ? kobject_put
[<>] ? trace_hardirqs_off_thunk
[<>] error_code
[<>] ? __mode_timer
[<>] ? do_invalid_op
[<>] ? ccid3_hc_tx_no_feedback_timer
[<>] run_timer_softirq
[<>] ? ccid3_hc_tx_no_feedback_timer
[<>] ? ccid3_hc_tx_no_feedback_timer
[<>] __do_softirq
[<>] ? __do_softirq
<IRQ> [<>] ? irq_exit
[<>] ? smp_apic_timer_interrupt
[<>] ? apic_timer_interrupt
[<>] ? native_safe_halt
[<>] ? default_idle
[<>] ? cpu_idle
[<>] ? rest_init
[<>] ? start_kernel
[<>] ? i386_start_kernel

Thanks

 --
Burak



----- Original Message ----
> From: Gerrit Renker <gerrit@erg.abdn.ac.uk>
> To: Burak Gorkemli <burakgmail-dccp@yahoo.com>
> Cc: dccp@vger.kernel.org
> Sent: Saturday, June 6, 2009 9:01:53 PM
> Subject: Re: yet another panic on 2.6.30-rc6
> 
> Hi Burak,
> 
> thank you for raising this. We need to isolate the cause of the problem.
> 
> The dccp test tree is based on the netdev-2.6 tree maintained by Dave
> Miller, which itself contains other trees (I think the total number of
> kernel trees is ~130).
> 
> I am testing this tree myself, though mostly with audio streaming over
> WiFi links, using the paraslash streamer.
> 
> From looking through the trace it looks as though this is a timer
> problem. Any chance of getting the EIP value to see where it 
> 
> Regarding reproducability -- unfortunately I had just put in a new
> revision onto eden-feed. Can you please consider pulling again from
> 
>     git://eden-feed.erg.abdn.ac.uk/dccp_exp [subtrees dccp/ccid4]
> 
> If this problem persists, it would be good to CC: netdev@vger as well.
> 
> It is quite possible that the problem may have cleared itself as a
> result of using more recent sources. The dccp test tree gets a fresh
> base at least every week, to avoid transient problems.
> 
> Thanks in advance
> Gerrit
> 
> 
> Quoting Burak Gorkemli:
> |
> | The console output for this one is as follows:
> |
> | [<>] do_trap
> | [<>] ? do_invalid_op
> | [<>] do_invalid_op
> | [<>] ? ccid3_hc_tx_no_feedback_timer
> | [<>] ? e1000_clean_rx_irq
> | [<>] ? trace_hardirqs_off_thunk
> | [<>] error_code
> | [<>] ? __mod_timer
> | [<>] ? do_invalid_op
> | [<>] ? ccid3_hc_tx_no_feedback_timer
> | [<>] run_timer_softirq
> | [<>] ? ccid3_hc_tx_no_feedback_timer
> | [<>] ? ccid3_hc_tx_no_feedback_timer
> | [<>] __do_softirq
> | [<>] ? __do_softirq
> | [<>] ? irq_exit
> | [<>] ? smp_apic_timer_interrupt
> | [<>] ? apic_timer_interrupt
> | [<>] ? native_safe_halt
> | [<>] ? default_idle
> | [<>] ? cpu_idle
> | [<>] ? rest_init
> | [<>] ? start_kernel
> | [<>] ? i386_start_kernel
> |
> |  --
> | Burak
> |
> |
> |
> | ----- Original Message ----
> | > From: Burak Gorkemli 
> | > To: dccp@vger.kernel.org
> | > Sent: Saturday, June 6, 2009 1:58:59 AM
> | > Subject: panic on 2.6.30-rc6
> | >
> | >
> | > Hi,
> | >
> | > I am having kernel panic while doing video streaming tests with ccid3, using
> | > kernel version 2.6.30-rc6, pulled from Gerrit's
> | > tree. The crash happens on the receiver node (there is a sender node and a
> | > receiver node, both Linux boxes running 2.6.30-rc6) and it is quite
> | > reproducible. The console output is as follows:
> | >
> | > [<>] ? do_invalid_op
> | > [<>] do_invalid_op
> | > [<>] ? ccid3_hc_tx_no_feedback_timer
> | > [<>] ? __wake_up
> | > [<>] ? __kfree_skb
> | > [<>] ? trace_hardirqs_off_thunk
> | > [<>] error_code
> | > [<>] ? __mod_timer
> | > [<>] ? do_invalid_op
> | > [<>] ? ccid3_hc_tx_no_feedback_timer
> | > [<>] run_timer_softirq
> | > [<>] ? ccid3_hc_tx_no_feedback_timer
> | > [<>] ? ccid3_hc_tx_no_feedback_timer
> | > [<>] __do_softirq
> | > [<>] ? __do_softirq
> | > [<>] ? irq_exit
> | > [<>] ? smp_apic_timer_interrupt
> | > [<>] ? apic_timer_interrupt
> | > [<>] ? native_safe_halt
> | > [<>] ? default_idle
> | > [<>] ? cpu_idle
> | > [<>] ? rest_init
> | > [<>] ? start_kernel
> | > [<>] ? i386_start_kernel
> | >
> | > Any comments?
> --
> To unsubscribe from this list: send the line "unsubscribe dccp" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: yet another panic on 2.6.30-rc6

[Burak Gorkemli]

Hi Gerrit,

Sorry for the late reply, I have just found the time today to pull the latest
tree from your site and test it. It still crashes unfortunately, giving similar
console output, as given below. As for the transient problems, I had the
same issue with an older kernel, 2.6.26-rc5, again pulled from your site.

BTW, how to print EIP after the panic?

The console output of the latest panic is as follows:

[<>] ? ccid3_hc_tx_no_feedback_timer
[<>] ? default_spin_lock_flags
[<>] ? elv_queue_empty
[<>] ? blk_run_queue
[<>] ? kobject_put
[<>] ? trace_hardirqs_off_thunk
[<>] error_code
[<>] ? __mode_timer
[<>] ? do_invalid_op
[<>] ? ccid3_hc_tx_no_feedback_timer
[<>] run_timer_softirq
[<>] ? ccid3_hc_tx_no_feedback_timer
[<>] ? ccid3_hc_tx_no_feedback_timer
[<>] __do_softirq
[<>] ? __do_softirq
<IRQ> [<>] ? irq_exit
[<>] ? smp_apic_timer_interrupt
[<>] ? apic_timer_interrupt
[<>] ? native_safe_halt
[<>] ? default_idle
[<>] ? cpu_idle
[<>] ? rest_init
[<>] ? start_kernel
[<>] ? i386_start_kernel

Thanks

 --
Burak



----- Original Message ----
> From: Gerrit Renker <gerrit@erg.abdn.ac.uk>
> To: Burak Gorkemli <burakgmail-dccp@yahoo.com>
> Cc: dccp@vger.kernel.org
> Sent: Saturday, June 6, 2009 9:01:53 PM
> Subject: Re: yet another panic on 2.6.30-rc6
> 
> Hi Burak,
> 
> thank you for raising this. We need to isolate the cause of the problem.
> 
> The dccp test tree is based on the netdev-2.6 tree maintained by Dave
> Miller, which itself contains other trees (I think the total number of
> kernel trees is ~130).
> 
> I am testing this tree myself, though mostly with audio streaming over
> WiFi links, using the paraslash streamer.
> 
> From looking through the trace it looks as though this is a timer
> problem. Any chance of getting the EIP value to see where it 
> 
> Regarding reproducability -- unfortunately I had just put in a new
> revision onto eden-feed. Can you please consider pulling again from
> 
>     git://eden-feed.erg.abdn.ac.uk/dccp_exp [subtrees dccp/ccid4]
> 
> If this problem persists, it would be good to CC: netdev@vger as well.
> 
> It is quite possible that the problem may have cleared itself as a
> result of using more recent sources. The dccp test tree gets a fresh
> base at least every week, to avoid transient problems.
> 
> Thanks in advance
> Gerrit
> 
> 
> Quoting Burak Gorkemli:
> |
> | The console output for this one is as follows:
> |
> | [<>] do_trap
> | [<>] ? do_invalid_op
> | [<>] do_invalid_op
> | [<>] ? ccid3_hc_tx_no_feedback_timer
> | [<>] ? e1000_clean_rx_irq
> | [<>] ? trace_hardirqs_off_thunk
> | [<>] error_code
> | [<>] ? __mod_timer
> | [<>] ? do_invalid_op
> | [<>] ? ccid3_hc_tx_no_feedback_timer
> | [<>] run_timer_softirq
> | [<>] ? ccid3_hc_tx_no_feedback_timer
> | [<>] ? ccid3_hc_tx_no_feedback_timer
> | [<>] __do_softirq
> | [<>] ? __do_softirq
> | [<>] ? irq_exit
> | [<>] ? smp_apic_timer_interrupt
> | [<>] ? apic_timer_interrupt
> | [<>] ? native_safe_halt
> | [<>] ? default_idle
> | [<>] ? cpu_idle
> | [<>] ? rest_init
> | [<>] ? start_kernel
> | [<>] ? i386_start_kernel
> |
> |  --
> | Burak
> |
> |
> |
> | ----- Original Message ----
> | > From: Burak Gorkemli 
> | > To: dccp@vger.kernel.org
> | > Sent: Saturday, June 6, 2009 1:58:59 AM
> | > Subject: panic on 2.6.30-rc6
> | >
> | >
> | > Hi,
> | >
> | > I am having kernel panic while doing video streaming tests with ccid3, using
> | > kernel version 2.6.30-rc6, pulled from Gerrit's
> | > tree. The crash happens on the receiver node (there is a sender node and a
> | > receiver node, both Linux boxes running 2.6.30-rc6) and it is quite
> | > reproducible. The console output is as follows:
> | >
> | > [<>] ? do_invalid_op
> | > [<>] do_invalid_op
> | > [<>] ? ccid3_hc_tx_no_feedback_timer
> | > [<>] ? __wake_up
> | > [<>] ? __kfree_skb
> | > [<>] ? trace_hardirqs_off_thunk
> | > [<>] error_code
> | > [<>] ? __mod_timer
> | > [<>] ? do_invalid_op
> | > [<>] ? ccid3_hc_tx_no_feedback_timer
> | > [<>] run_timer_softirq
> | > [<>] ? ccid3_hc_tx_no_feedback_timer
> | > [<>] ? ccid3_hc_tx_no_feedback_timer
> | > [<>] __do_softirq
> | > [<>] ? __do_softirq
> | > [<>] ? irq_exit
> | > [<>] ? smp_apic_timer_interrupt
> | > [<>] ? apic_timer_interrupt
> | > [<>] ? native_safe_halt
> | > [<>] ? default_idle
> | > [<>] ? cpu_idle
> | > [<>] ? rest_init
> | > [<>] ? start_kernel
> | > [<>] ? i386_start_kernel
> | >
> | > Any comments?
> --
> To unsubscribe from this list: send the line "unsubscribe dccp" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: yet another panic on 2.6.30-rc6

[Gerrit Renker]

Hi Burak,

it would be good to have more information, how this bug can be triggered:

 * what is the hardware,
 * can it be reproduced in the same manner on other hardware,
 * with which configuration (.config file),
 * does it only happen with CCID 3? Here it is good to know whether
   - Tickless System (CONFIG_NO_HZ) or
   - High-resolution timers (CONFIG_HIGH_RES_TIMERS)
   are enabled,
 * enabling kernel debugging features (below).

Further - are you running DCCP in a virtualised environment?

I checked some of the functions below: "default_spin_lock_flags" is defined
in arch/x86/kernel/paravirt-spinlocks.c. When running a network protocol
that is designed for longer distance networks, some underlying assumptions
may no longer hold in the same way as designed.

| BTW, how to print EIP after the panic?
It needs to be enabled in the kernel configuration. Please enable

 1/ "General setup"  -> "BUG() support"    (CONFIG_BUG=y)
 2/ "Kernel hacking" -> "Kernel debugging" (CONFIG_DEBUG_KERNEL=y)
 3/ "Kernel hacking" (depends on (2) ->
    "Verbose BUG() reporting (adds 70k)"   (CONFIG_DEBUG_BUGVERBOSE=y)
 4/ When enabling (2), it also can not harm to enable the

| The console output of the latest panic is as follows:
|
| [<>] ? ccid3_hc_tx_no_feedback_timer
| [<>] ? default_spin_lock_flags
| [<>] ? elv_queue_empty
| [<>] ? blk_run_queue
| [<>] ? kobject_put
| [<>] ? trace_hardirqs_off_thunk
| [<>] error_code
| [<>] ? __mode_timer
| [<>] ? do_invalid_op
| [<>] ? ccid3_hc_tx_no_feedback_timer
| [<>] run_timer_softirq
| [<>] ? ccid3_hc_tx_no_feedback_timer
| [<>] ? ccid3_hc_tx_no_feedback_timer
| [<>] __do_softirq
| [<>] ? __do_softirq
| <IRQ> [<>] ? irq_exit
| [<>] ? smp_apic_timer_interrupt
| [<>] ? apic_timer_interrupt
| [<>] ? native_safe_halt
| [<>] ? default_idle
| [<>] ? cpu_idle
| [<>] ? rest_init
| [<>] ? start_kernel
| [<>] ? i386_start_kernel

Re: yet another panic on 2.6.30-rc6

[Gerrit Renker]

Hi Burak,

it would be good to have more information, how this bug can be triggered:

 * what is the hardware,
 * can it be reproduced in the same manner on other hardware,
 * with which configuration (.config file),
 * does it only happen with CCID 3? Here it is good to know whether
   - Tickless System (CONFIG_NO_HZ) or
   - High-resolution timers (CONFIG_HIGH_RES_TIMERS)
   are enabled,
 * enabling kernel debugging features (below).

Further - are you running DCCP in a virtualised environment?

I checked some of the functions below: "default_spin_lock_flags" is defined
in arch/x86/kernel/paravirt-spinlocks.c. When running a network protocol
that is designed for longer distance networks, some underlying assumptions
may no longer hold in the same way as designed.

| BTW, how to print EIP after the panic?
It needs to be enabled in the kernel configuration. Please enable

 1/ "General setup"  -> "BUG() support"    (CONFIG_BUG=y)
 2/ "Kernel hacking" -> "Kernel debugging" (CONFIG_DEBUG_KERNEL=y)
 3/ "Kernel hacking" (depends on (2) ->
    "Verbose BUG() reporting (adds 70k)"   (CONFIG_DEBUG_BUGVERBOSE=y)
 4/ When enabling (2), it also can not harm to enable the

| The console output of the latest panic is as follows:
|
| [<>] ? ccid3_hc_tx_no_feedback_timer
| [<>] ? default_spin_lock_flags
| [<>] ? elv_queue_empty
| [<>] ? blk_run_queue
| [<>] ? kobject_put
| [<>] ? trace_hardirqs_off_thunk
| [<>] error_code
| [<>] ? __mode_timer
| [<>] ? do_invalid_op
| [<>] ? ccid3_hc_tx_no_feedback_timer
| [<>] run_timer_softirq
| [<>] ? ccid3_hc_tx_no_feedback_timer
| [<>] ? ccid3_hc_tx_no_feedback_timer
| [<>] __do_softirq
| [<>] ? __do_softirq
| <IRQ> [<>] ? irq_exit
| [<>] ? smp_apic_timer_interrupt
| [<>] ? apic_timer_interrupt
| [<>] ? native_safe_halt
| [<>] ? default_idle
| [<>] ? cpu_idle
| [<>] ? rest_init
| [<>] ? start_kernel
| [<>] ? i386_start_kernel