System becomes unresponsive due to kernel oops (IP: dev_queue_xmit+0x256/0x3f4)

System becomes unresponsive due to kernel oops (IP: dev_queue_xmit+0x256/0x3f4)

[prasad zambare]

[-- Attachment #1: Type: text/plain, Size: 698 bytes --]

Hi All,

From past few days I am trying to debug an issue but I am not able to
get a break through.

Linux system is consistently getting unresponsive with below serial
console output. The similar serial console output is observed every
time the issue is occurred.

Steps to reproduce this issue are unknown to me. But, this issue is
not observed when all parameters related to acpi are disabled from
BIOS.

I am newbie to debugging kernel oops. I am using Fedora 13 and I can
not upgrade to latest release. Please let me know what could be the
problem and how can I resolve this issue. Any pointer or help will be
of great importance.

Please find the kernel oops message below

Thank You,
Prasad

[-- Attachment #2: stack_trace3 --]
[-- Type: application/octet-stream, Size: 4991 bytes --]

	BUG: unable to handle kernel NULL pointer dereference at (null)
	IP: [<c06fbcdd>] dev_queue_xmit+0x256/0x3f4
	*pdpt = 000000002ecb3001 *pde = 000000012974c067 
	Oops: 0000 [#1] SMP 
	last sysfs file: /sys/devices/pci0000:00/0000:00:1c.0/0000:01:00.0/net/eth0/broadcast
	Modules linked in: tun nfnetlink_queue nfnetlink bluetooth rfkill ts_kmp xt_string 8021q garp nf_nat_pptp nf_conntrack_pptp nf_conntrack_proto_]

	Pid: 14113, comm: snort Not tainted 2.6.33.3-85.fc13.i686.PAE #1 To be filled by O.E.M./To Be Filled By O.E.M.
	EIP: 0060:[<c06fbcdd>] EFLAGS: 00210202 CPU: 1
	EIP is at dev_queue_xmit+0x256/0x3f4
	EAX: f6922000 EBX: f6bf5a80 ECX: ed524140 EDX: f6123380
	ESI: f6248000 EDI: 00000000 EBP: eef7dbf0 ESP: eef7dbdc
	 DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
	Process snort (pid: 14113, ti=eef7c000 task=eef28cc0 task.ti=eef7c000)
	Stack:
	 eef7dbec f6923300 f6bf5a80 ef48f360 ed524108 eef7dc14 c0722500 001aa8b4
	<0> 00000000 ef48f350 ef48f300 00000028 f6bf5a80 001aa8b4 eef7dc20 c0722591
	<0> f6bf5a80 eef7dc2c c0722819 f0fc4800 eef7dc34 c07216d5 eef7dc4c c0713b12
	Call Trace:
	 [<c0722500>] ? ip_finish_output2+0x18e/0x1c6
	 [<c0722591>] ? ip_finish_output+0x59/0x5c
	 [<c0722819>] ? ip_output+0x74/0x79
	 [<c07216d5>] ? dst_output+0x9/0xb
	 [<c0713b12>] ? nf_reinject+0xa3/0xe6
	 [<f80ab427>] ? nfqnl_recv_verdict+0x1cf/0x1e0 [nfnetlink_queue]
	 [<f7e6b1ab>] ? nfnetlink_rcv_msg+0x118/0x149 [nfnetlink]
	 [<f7e6b0b9>] ? nfnetlink_rcv_msg+0x26/0x149 [nfnetlink]
	 [<c0711903>] ? netlink_sendmsg+0x72/0x221
	 [<f7e6b093>] ? nfnetlink_rcv_msg+0x0/0x149 [nfnetlink]
	 [<c0711130>] ? netlink_rcv_skb+0x30/0x76
	 [<f7e6b08c>] ? nfnetlink_rcv+0x1b/0x22 [nfnetlink]
	 [<c0710f6f>] ? netlink_unicast+0xbe/0x119
	 [<c0711aa5>] ? netlink_sendmsg+0x214/0x221
	 [<c06edfad>] ? __sock_sendmsg+0x45/0x4e
	 [<c06ee254>] ? sock_sendmsg+0x93/0xa7
	 [<c0442bfc>] ? irq_exit+0x39/0x5c
	 [<c0409c05>] ? do_IRQ+0x86/0x9a
	 [<c0408df0>] ? common_interrupt+0x30/0x38
	 [<c06f625f>] ? verify_iovec+0x57/0x6c
	 [<c06ee676>] ? sys_sendmsg+0x187/0x1eb
	 [<c06ee4c2>] ? sockfd_lookup_light+0x16/0x43
	 [<c06ee4aa>] ? fput_light+0xc/0xe
	 [<c06ef6d7>] ? sys_recvfrom+0x102/0x121
	 [<c06fbf04>] ? dev_kfree_skb_any+0x27/0x32
	 [<f88c3dfb>] ? e1000_put_txbuf+0x50/0x65 [e1000e]
	 [<f88c3ee8>] ? e1000_clean_tx_irq+0xa7/0x1dc [e1000e]
	 [<c05a6680>] ? might_fault+0x19/0x1b
	 [<c05a68eb>] ? copy_to_user+0x2f/0x108
	 [<c05a6680>] ? might_fault+0x19/0x1b
	 [<c06efe80>] ? sys_socketcall+0x15e/0x1a5
	 [<c040ff01>] ? syscall_trace_leave+0xa5/0xb8
	 [<c0782bdc>] ? syscall_call+0x7/0xb
	 [<c0780000>] ? acpi_processor_add+0x1f/0x74b
	Code: 57 0c 66 89 83 80 00 00 00 8b 96 00 02 00 00 0f b7 c0 c1 e0 07 01 d0 89 45 f0 8b 78 04 66 8b 43 7e 80 e4 cf 80 cc 20 66 89 43 7e <83> 3f  
	EIP: [<c06fbcdd>] dev_queue_xmit+0x256/0x3f4 SS:ESP 0068:eef7dbdc
	CR2: 0000000000000000
	---[ end trace 5e9db4f99c9e9021 ]---

	Kernel panic - not syncing: Fatal exception in interrupt
	Message from syslogd@machine Pid: 14113, comm: snort Tainted: G      D    2.6.33.3-85.fc13.i686.PAE #1
	Call Trace:
	 [<c0780b4f>] ? printk+0xf/0x18
	 [<c0780a8d>] panic+0x39/0xec
	 [<c0783c90>] oops_end+0x92/0xa1
	 [<c04261c1>] no_context+0x13e/0x148
	 [<c04262b7>] __bad_area_nosemaphore+0xec/0xf4
	 [<c0784e87>] ? do_page_fault+0x0/0x2fa
	 [<c04262cc>] bad_area_nosemaphore+0xd/0x10
	 [<c078501b>] do_page_fault+0x194/0x2fa
	 [<c0784e87>] ? do_page_fault+0x0/0x2fa
	 [<c07832df>] error_code+0x73/0x78
	 [<c06fbcdd>] ? dev_queue_xmit+0x256/0x3f4
	 [<c0722500>] ip_finish_output2+0x18e/0x1c6
	 [<c0722591>] ip_finish_output+0x59/0x5c
	 [<c0722819>] ip_output+0x74/0x79
	 [<c07216d5>] dst_output+0x9/0xb
	 [<c0713b12>] nf_reinject+0xa3/0xe6
	 [<f80ab427>] nfqnl_recv_verdict+0x1cf/0x1e0 [nfnetlink_queue]
	 [<f7e6b1ab>] nfnetlink_rcv_msg+0x118/0x149 [nfnetlink]
	 [<f7e6b0b9>] ? nfnetlink_rcv_msg+0x26/0x149 [nfnetlink]
	 [<c0711903>] ? netlink_sendmsg+0x72/0x221
	 [<f7e6b093>] ? nfnetlink_rcv_msg+0x0/0x149 [nfnetlink]
	 [<c0711130>] netlink_rcv_skb+0x30/0x76
	 [<f7e6b08c>] nfnetlink_rcv+0x1b/0x22 [nfnetlink]
	 [<c0710f6f>] netlink_unicast+0xbe/0x119
	 [<c0711aa5>] netlink_sendmsg+0x214/0x221
	 [<c06edfad>] __sock_sendmsg+0x45/0x4e
	 [<c06ee254>] sock_sendmsg+0x93/0xa7
	 [<c0442bfc>] ? irq_exit+0x39/0x5c
	 [<c0409c05>] ? do_IRQ+0x86/0x9a
	 [<c0408df0>] ? common_interrupt+0x30/0x38
	 [<c06f625f>] ? verify_iovec+0x57/0x6c
	 [<c06ee676>] sys_sendmsg+0x187/0x1eb
	 [<c06ee4c2>] ? sockfd_lookup_light+0x16/0x43
	 [<c06ee4aa>] ? fput_light+0xc/0xe
	 [<c06ef6d7>] ? sys_recvfrom+0x102/0x121
	 [<c06fbf04>] ? dev_kfree_skb_any+0x27/0x32
	 [<f88c3dfb>] ? e1000_put_txbuf+0x50/0x65 [e1000e]
	 [<f88c3ee8>] ? e1000_clean_tx_irq+0xa7/0x1dc [e1000e]
	 [<c05a6680>] ? might_fault+0x19/0x1b
	 [<c05a68eb>] ? copy_to_user+0x2f/0x108
	 [<c05a6680>] ? might_fault+0x19/0x1b
	 [<c06efe80>] sys_socketcall+0x15e/0x1a5
	 [<c040ff01>] ? syscall_trace_leave+0xa5/0xb8
	 [<c0782bdc>] syscall_call+0x7/0xb
	 [<c0780000>] ? acpi_processor_add+0x1f/0x74b

Re: System becomes unresponsive due to kernel oops (IP: dev_queue_xmit+0x256/0x3f4)

[Marcelo Ricardo Leitner]

On 01-11-2014 11:44, prasad zambare wrote:
> Hi All,
>
>  From past few days I am trying to debug an issue but I am not able to
> get a break through.
>
> Linux system is consistently getting unresponsive with below serial
> console output. The similar serial console output is observed every
> time the issue is occurred.
>
> Steps to reproduce this issue are unknown to me. But, this issue is
> not observed when all parameters related to acpi are disabled from
> BIOS.
>
> I am newbie to debugging kernel oops. I am using Fedora 13 and I can
> not upgrade to latest release. Please let me know what could be the
> problem and how can I resolve this issue. Any pointer or help will be
> of great importance.
>
> Please find the kernel oops message below
>
> Thank You,
> Prasad

If you can't upgrade, are you willing/able to deploy a custom kernel with a 
fix? Otherwise I'm afraid there isn't much to do in here.

Marcelo

Re: System becomes unresponsive due to kernel oops (IP: dev_queue_xmit+0x256/0x3f4)

[prasad zambare]

HI Marcelo,

Thank you for your reply.

I am ready deploy custom kernel. Today itself I have started to work
on this. I have prepared 2.6.39 kernel rpm and installed it on the
system.When observed that network interfaces are using e1000e deriver
(by ethtool -i <interface name> command), it is also updated to latest
version. Tomorrow, I am going to see if this resolves the issue.

I guess using custom kernel or upgrading related packages is the only
available option. But this is only trial and error approach as exact
reproduction steps are unknown. Please share if there are any pointers
to understand this issue better or for its resolution.

Thank You,
Prasad

On Tue, Nov 4, 2014 at 9:30 PM, Marcelo Ricardo Leitner
<marcelo.leitner@gmail.com> wrote:
> On 01-11-2014 11:44, prasad zambare wrote:
>>
>> Hi All,
>>
>>  From past few days I am trying to debug an issue but I am not able to
>> get a break through.
>>
>> Linux system is consistently getting unresponsive with below serial
>> console output. The similar serial console output is observed every
>> time the issue is occurred.
>>
>> Steps to reproduce this issue are unknown to me. But, this issue is
>> not observed when all parameters related to acpi are disabled from
>> BIOS.
>>
>> I am newbie to debugging kernel oops. I am using Fedora 13 and I can
>> not upgrade to latest release. Please let me know what could be the
>> problem and how can I resolve this issue. Any pointer or help will be
>> of great importance.
>>
>> Please find the kernel oops message below
>>
>> Thank You,
>> Prasad
>
>
> If you can't upgrade, are you willing/able to deploy a custom kernel with a
> fix? Otherwise I'm afraid there isn't much to do in here.
>
> Marcelo
>

Re: System becomes unresponsive due to kernel oops (IP: dev_queue_xmit+0x256/0x3f4)

[prasad zambare]

Hi All,

With 2.6.39 kernel and e1000e I had no luck and again the system gets
unresponsive. This time the change is I got "Kernel: Disabling IRQ
#57" message which is repeated for numbers 51, 48, 54. cat
/proc/interrupts shows these numbers are associated with configured
network interfaces. This time I did not get serial console logs as I
made a mistake while connecting serial console. Tomorrow I will again
try to reproduce this issue with proper serial console setup.

Now I not sure if this is the same issue or I have hit a new one.
Please share your views on this tryout.

Thank You,
Prasad

On Tue, Nov 4, 2014 at 11:50 PM, prasad zambare <prasadzambare@gmail.com> wrote:
> HI Marcelo,
>
> Thank you for your reply.
>
> I am ready deploy custom kernel. Today itself I have started to work
> on this. I have prepared 2.6.39 kernel rpm and installed it on the
> system.When observed that network interfaces are using e1000e deriver
> (by ethtool -i <interface name> command), it is also updated to latest
> version. Tomorrow, I am going to see if this resolves the issue.
>
> I guess using custom kernel or upgrading related packages is the only
> available option. But this is only trial and error approach as exact
> reproduction steps are unknown. Please share if there are any pointers
> to understand this issue better or for its resolution.
>
> Thank You,
> Prasad
>
> On Tue, Nov 4, 2014 at 9:30 PM, Marcelo Ricardo Leitner
> <marcelo.leitner@gmail.com> wrote:
>> On 01-11-2014 11:44, prasad zambare wrote:
>>>
>>> Hi All,
>>>
>>>  From past few days I am trying to debug an issue but I am not able to
>>> get a break through.
>>>
>>> Linux system is consistently getting unresponsive with below serial
>>> console output. The similar serial console output is observed every
>>> time the issue is occurred.
>>>
>>> Steps to reproduce this issue are unknown to me. But, this issue is
>>> not observed when all parameters related to acpi are disabled from
>>> BIOS.
>>>
>>> I am newbie to debugging kernel oops. I am using Fedora 13 and I can
>>> not upgrade to latest release. Please let me know what could be the
>>> problem and how can I resolve this issue. Any pointer or help will be
>>> of great importance.
>>>
>>> Please find the kernel oops message below
>>>
>>> Thank You,
>>> Prasad
>>
>>
>> If you can't upgrade, are you willing/able to deploy a custom kernel with a
>> fix? Otherwise I'm afraid there isn't much to do in here.
>>
>> Marcelo
>>

Re: System becomes unresponsive due to kernel oops (IP: dev_queue_xmit+0x256/0x3f4)

[Marcelo Ricardo Leitner]

On 05-11-2014 14:33, prasad zambare wrote:
> Hi All,
>
> With 2.6.39 kernel and e1000e I had no luck and again the system gets
> unresponsive. This time the change is I got "Kernel: Disabling IRQ
> #57" message which is repeated for numbers 51, 48, 54. cat
> /proc/interrupts shows these numbers are associated with configured
> network interfaces. This time I did not get serial console logs as I
> made a mistake while connecting serial console. Tomorrow I will again
> try to reproduce this issue with proper serial console setup.
>
> Now I not sure if this is the same issue or I have hit a new one.
> Please share your views on this tryout.

If you're deploying a custom kernel, what stops you from trying a newer 
one, like 3.17? :)

Marcelo