* 2.6.32-rc4: Reported regressions from 2.6.31
@ 2009-10-11 22:07 Rafael J. Wysocki
[not found] ` <r-4Wjt4coy.A.TPD.dal0KB@chimera>
[not found] ` <r-4Wjt4coy.A.CwC.Wal0KB@chimera>
0 siblings, 2 replies; 12+ messages in thread
From: Rafael J. Wysocki @ 2009-10-11 22:07 UTC (permalink / raw)
To: Linux Kernel Mailing List
Cc: Adrian Bunk, DRI, Linux SCSI List, Network Development,
Linux Wireless List, Natalie Protasevich, Linux ACPI,
Andrew Morton, Kernel Testers List, Linus Torvalds, Linux PM List
[Note:
We started to get more regression reports for 2.6.32-rc (100%+ jump in the
last 10 days. Nevertheless, we're still receiving new reports of regressions
from 2.6.30.]
This message contains a list of some regressions from 2.6.31, for which there
are no fixes in the mainline I know of. If any of them have been fixed already,
please let me know.
If you know of any other unresolved regressions from 2.6.31, please let me know
either and I'll add them to the list. Also, please let me know if any of the
entries below are invalid.
Each entry from the list will be sent additionally in an automatic reply to
this message with CCs to the people involved in reporting and handling the
issue.
Listed regressions statistics:
Date Total Pending Unresolved
----------------------------------------
2009-10-12 48 31 27
2009-10-02 22 15 9
Unresolved regressions
----------------------
Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14392
Subject : Touchpad "paste" stops working after suspend to RAM
Submitter : Carlos R. Mafra <crmafra2@gmail.com>
Date : 2009-10-11 16:21 (1 days old)
References : http://marc.info/?l=linux-kernel&m=125527987316493&w=4
Handled-By : Dmitry Torokhov <dmitry.torokhov@gmail.com>
Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14390
Subject : "bind" a device to a driver doesn't not work anymore
Submitter : Éric Piel <Eric.Piel@tremplin-utc.net>
Date : 2009-10-11 0:04 (1 days old)
References : http://marc.info/?l=linux-kernel&m=125521979921241&w=4
Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14389
Subject : Build system issue
Submitter : Peter Zijlstra <peterz@infradead.org>
Date : 2009-10-09 8:58 (3 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=575543347b5baed0ca927cb90ba8807396fe9cc9
References : http://marc.info/?l=linux-kernel&m=125507914909152&w=4
Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14387
Subject : deadlock with fallocate
Submitter : Thomas Neumann <tneumann@users.sourceforge.net>
Date : 2009-10-07 3:00 (5 days old)
References : http://marc.info/?l=linux-kernel&m=125488495526471&w=4
Handled-By : Christoph Hellwig <hch@lst.de>
Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14386
Subject : GPF in snd_hda_intel
Submitter : Luca Tettamanti <kronos.it@gmail.com>
Date : 2009-10-10 13:01 (2 days old)
References : http://marc.info/?l=linux-kernel&m=125517999408019&w=4
Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14384
Subject : tbench regression with 2.6.32-rc1
Submitter : Zhang, Yanmin <yanmin_zhang@linux.intel.com>
Date : 2009-10-09 9:51 (3 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=59abf02644c45f1591e1374ee7bb45dc757fcb88
References : http://marc.info/?l=linux-kernel&m=125508216713138&w=4
Handled-By : Peter Zijlstra <a.p.zijlstra@chello.nl>
Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14383
Subject : hackbench regression with kernel 2.6.32-rc1
Submitter : Zhang, Yanmin <yanmin_zhang@linux.intel.com>
Date : 2009-10-09 9:19 (3 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=29cd8bae396583a2ee9a3340db8c5102acf9f6fd
References : http://marc.info/?l=linux-kernel&m=125508007510274&w=4
Handled-By : Peter Zijlstra <a.p.zijlstra@chello.nl>
Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14381
Subject : iwlagn lost connection after s2ram (with warnings)
Submitter : Carlos R. Mafra <crmafra2@gmail.com>
Date : 2009-10-07 14:20 (5 days old)
References : http://marc.info/?l=linux-kernel&m=125492569119947&w=4
Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14380
Subject : Video tearing/glitching with T400 laptops
Submitter : Theodore Ts'o <tytso@mit.edu>
Date : 2009-10-02 22:40 (10 days old)
References : http://marc.info/?l=linux-kernel&m=125452324520623&w=4
Handled-By : Jesse Barnes <jbarnes@virtuousgeek.org>
Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14379
Subject : ACPI Warning for _SB_.BAT0._BIF: Converted Buffer to expected String
Submitter : Justin Mattock <justinmattock@gmail.com>
Date : 2009-10-08 21:46 (4 days old)
References : http://marc.info/?l=linux-kernel&m=125504031328941&w=4
Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14378
Subject : Problems with net/core/skbuff.c
Submitter : Massimo Cetra <mcetra@navynet.it>
Date : 2009-10-08 14:51 (4 days old)
References : http://marc.info/?l=linux-kernel&m=125501488220358&w=4
Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14376
Subject : Kernel NULL pointer dereference/ kvm subsystem
Submitter : Don Dupuis <dondster@gmail.com>
Date : 2009-10-06 14:38 (6 days old)
References : http://marc.info/?l=linux-kernel&m=125484025021737&w=4
Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14374
Subject : MCEs caused by commit db8be50c4307dac2b37305fc59c8dc0f978d09ea
Submitter : Nick Piggin <npiggin@suse.de>
Date : 2009-10-02 7:34 (10 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=db8be50c4307dac2b37305fc59c8dc0f978d09ea
References : http://marc.info/?l=linux-kernel&m=125446885705223&w=4
Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14373
Subject : Task blocked for more than 120 seconds
Submitter : Zeno Davatz <zdavatz@gmail.com>
Date : 2009-10-02 10:16 (10 days old)
References : http://marc.info/?l=linux-kernel&m=125447858618412&w=4
Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14372
Subject : Wireless not working after suspend-resume
Submitter : Fabio Comolli <fabio.comolli@gmail.com>
Date : 2009-10-03 15:36 (9 days old)
References : http://lkml.org/lkml/2009/10/3/91
Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14370
Subject : ext4 corruptions
Submitter : Alexey Fisher <bug-track@fisher-privat.net>
Date : 2009-10-09 19:20 (3 days old)
References : http://marc.info/?l=linux-kernel&m=125511643504864&w=4
Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14355
Subject : USB serial regression after 2.6.31.1 with Huawei E169 GSM modem
Submitter : Benjamin Herrenschmidt <benh@kernel.crashing.org>
Date : 2009-10-10 03:07 (2 days old)
References : http://marc.info/?l=linux-kernel&m=125513456327542&w=4
Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14354
Subject : Bad corruption with 2.6.32-rc1 and upwards
Submitter : Holger Freyther <zecke@selfish.org>
Date : 2009-10-09 15:42 (3 days old)
Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14353
Subject : BUG: sleeping function called from invalid context at kernel/mutex.c:280
Submitter : Miles Lane <miles.lane@gmail.com>
Date : 2009-10-05 3:39 (7 days old)
References : http://marc.info/?l=linux-kernel&m=125471432208671&w=4
Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14352
Subject : WARNING: at net/mac80211/scan.c:267
Submitter : Maciej Rutecki <maciej.rutecki@gmail.com>
Date : 2009-10-08 00:30 (4 days old)
References : http://bugzilla.intellinuxwireless.org/show_bug.cgi?id=2089#c7
Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14334
Subject : pcmcia suspend regression from 2.6.31.1 to 2.6.31.2 - Dell Inspiron 600m
Submitter : Jose Marino <braket@hotmail.com>
Date : 2009-10-06 15:44 (6 days old)
Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14299
Subject : oops in wireless, iwl3945 related?
Submitter : Pavel Machek <pavel@ucw.cz>
Date : 2009-09-29 17:12 (13 days old)
References : http://marc.info/?l=linux-kernel&m=125424439725743&w=4
Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14298
Subject : warning at manage.c:361 (set_irq_wake), matrix-keypad related?
Submitter : Pavel Machek <pavel@ucw.cz>
Date : 2009-09-30 20:07 (12 days old)
References : http://marc.info/?l=linux-kernel&m=125434130703538&w=4
Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14297
Subject : console resume broken since ba15ab0e8d
Submitter : Sascha Hauer <s.hauer@pengutronix.de>
Date : 2009-09-30 15:11 (12 days old)
References : http://marc.info/?l=linux-kernel&m=125432349404060&w=4
Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14296
Subject : spitz boots but suspend/resume is broken
Submitter : Pavel Machek <pavel@ucw.cz>
Date : 2009-09-30 12:06 (12 days old)
References : http://marc.info/?l=linux-kernel&m=125431244516449&w=4
Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14279
Subject : Suspend to RAM freeze totally since 2.6.32-rc1 - Acer Aspire 1511Lmi laptop
Submitter : Christian Casteyde <casteyde.christian@free.fr>
Date : 2009-09-30 18:14 (12 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=5f68563996e812f9ca35b3939ad2a42e5d254d66
Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14277
Subject : Caught 8-bit read from freed memory in b43 driver at association
Submitter : Christian Casteyde <casteyde.christian@free.fr>
Date : 2009-09-30 18:06 (12 days old)
Regressions with patches
------------------------
Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14382
Subject : Transmit failure in et131x.
Submitter : Nick Bowler <nbowler@elliptictech.com>
Date : 2009-10-08 14:08 (4 days old)
References : http://marc.info/?l=linux-kernel&m=125501117713744&w=4
Handled-By : Alan Cox <alan@lxorguk.ukuu.org.uk>
Patch : http://patchwork.kernel.org/patch/52698/
Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14375
Subject : Intel(R) I/OAT DMA Engine init failed
Submitter : Alexander Beregalov <a.beregalov@gmail.com>
Date : 2009-10-02 9:46 (10 days old)
References : http://marc.info/?l=linux-kernel&m=125447680016160&w=4
Handled-By : Dan Williams <dan.j.williams@intel.com>
Patch : http://patchwork.kernel.org/patch/51808/
Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14302
Subject : Kernel panic on i386 machine when booting with profile=2
Submitter : Shi, Alex <alex.shi@intel.com>
Date : 2009-10-01 3:23 (11 days old)
References : http://marc.info/?l=linux-kernel&m=125436749607199&w=4
Handled-By : Alex Shi <alex.shi@intel.com>
Patch : http://patchwork.kernel.org/patch/50813/
Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14278
Subject : New message "NOHZ: local_softirq_pending 08" at each ping request
Submitter : Christian Casteyde <casteyde.christian@free.fr>
Date : 2009-09-30 18:12 (12 days old)
Handled-By : Michael Buesch <mb@bu3sch.de>
Patch : http://bugzilla.kernel.org/attachment.cgi?id=23220
For details, please visit the bug entries and follow the links given in
references.
As you can see, there is a Bugzilla entry for each of the listed regressions.
There also is a Bugzilla entry used for tracking the regressions from 2.6.31,
unresolved as well as resolved, at:
http://bugzilla.kernel.org/show_bug.cgi?id=14230
Please let me know if there are any Bugzilla entries that should be added to
the list in there.
Thanks,
Rafael
_______________________________________________
linux-pm mailing list
linux-pm@lists.linux-foundation.org
https://lists.linux-foundation.org/mailman/listinfo/linux-pm
^ permalink raw reply [flat|nested] 12+ messages in thread[parent not found: <r-4Wjt4coy.A.TPD.dal0KB@chimera>]
[parent not found: <b637ec0b0910121148g34f46481n1d5d4edb67b0ac55@mail.gmail.com>]
* Re: [Bug #14372] Wireless not working after suspend-resume [not found] ` <b637ec0b0910121148g34f46481n1d5d4edb67b0ac55@mail.gmail.com> @ 2009-10-12 21:55 ` Rafael J. Wysocki 0 siblings, 0 replies; 12+ messages in thread From: Rafael J. Wysocki @ 2009-10-12 21:55 UTC (permalink / raw) To: Fabio Comolli Cc: Linux Kernel Mailing List, Kernel Testers List, Johannes Berg, John W. Linville, netdev, Linux Wireless List On Monday 12 October 2009, Fabio Comolli wrote: > Still present with -rc4. > > Error is: > > [ 690.547660] wlan0: direct probe to AP 00:11:95:09:2c:e0 (try 1) > [ 690.550064] wlan0: direct probe responded > [ 690.550074] wlan0: authenticate with AP 00:11:95:09:2c:e0 (try 1) > [ 690.551629] wlan0: authenticated > [ 690.551663] wlan0: associate with AP 00:11:95:09:2c:e0 (try 1) > [ 690.553289] wlan0: RX AssocResp from 00:11:95:09:2c:e0 (capab=0x451 > status=12 aid=1) > [ 690.553298] wlan0: AP denied association (code=12) > > rfkill-ing the device off and then on fixes the problem. Thanks for the update. Rafael ^ permalink raw reply [flat|nested] 12+ messages in thread
[parent not found: <r-4Wjt4coy.A.CwC.Wal0KB@chimera>]
[parent not found: <20091012.034224.64980795.davem@davemloft.net>]
* Re: [Bug #14378] Problems with net/core/skbuff.c [not found] ` <20091012.034224.64980795.davem@davemloft.net> @ 2009-10-13 9:11 ` Massimo Cetra 2009-10-13 9:23 ` Massimo Cetra ` (2 more replies) 0 siblings, 3 replies; 12+ messages in thread From: Massimo Cetra @ 2009-10-13 9:11 UTC (permalink / raw) To: David Miller; +Cc: rjw, linux-kernel, kernel-testers, netdev [-- Attachment #1: Type: text/plain, Size: 959 bytes --] David Miller ha scritto: > From: "Rafael J. Wysocki" <rjw@sisk.pl> > Date: Mon, 12 Oct 2009 00:22:04 +0200 (CEST) > > >> Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14378 >> Subject : Problems with net/core/skbuff.c >> Submitter : Massimo Cetra <mcetra@navynet.it> >> Date : 2009-10-08 14:51 (4 days old) >> References : http://marc.info/?l=linux-kernel&m=125501488220358&w=4 >> > > I don't know what to do about this one. > > The user indicates that they have the vserver patches applied, > so maybe there is some interaction with that stuff. > Actually i found another oops which is very similar to the previous one. Here, vserver is not involved, and the problem starts at drbd which lives in kernel space (the other oops started at ocfs2). Both ocfs2 and drbd make heavy use of network I/O so i guess the problem is something in the network layer. Anything i can do to help to debugging and solving this issue ? Thanks Max [-- Attachment #2: oops2.txt --] [-- Type: text/plain, Size: 122916 bytes --] [60257.728500] ------------[ cut here ]------------ [60257.728500] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8() [60257.728500] Hardware name: [60257.728500] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys [60257.728500] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1 [60257.728500] Call Trace: [60257.728500] <IRQ> [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60257.728500] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60257.728500] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3 [60257.728500] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60257.728500] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d [60257.728500] [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net] [60257.728500] [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net] [60257.728500] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205 [60257.728500] [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole] [60257.728500] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f [60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60257.728500] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba [60257.728500] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b [60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60257.728500] [<ffffffff81308309>] ? printk+0x4e/0x5d [60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60257.728500] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf [60257.728500] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49 [60257.728500] [<ffffffff810626b7>] ? ktime_get+0xc/0x41 [60257.728500] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146 [60257.728500] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93 [60257.728500] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20 [60257.728500] <EOI> [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3 [60257.728500] [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31 [60257.728500] [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec [60257.728500] [<ffffffff81308f73>] ? thread_return+0x47/0xd5 [60257.728500] [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a [60257.728500] [<ffffffff81309213>] ? schedule_timeout+0x21/0x197 [60257.728500] [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10 [60257.728500] [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29 [60257.728500] [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3 [60257.728500] [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca [60257.728500] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e [60257.728500] [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac [60257.728500] [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd [60257.728500] [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5 [60257.728500] [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45 [60257.728500] [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d [60257.728500] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e [60257.728500] [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41 [60257.728500] [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd] [60257.728500] [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276 [60257.728500] [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd] [60257.728500] [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd] [60257.728500] [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd] [60257.728500] [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd] [60257.728500] [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd] [60257.728500] [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd] [60257.728500] [<ffffffff81011b9a>] ? child_rip+0xa/0x20 [60257.728500] [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4 [60257.728500] [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd] [60257.728500] [<ffffffff81011b90>] ? child_rip+0x0/0x20 [60257.728500] ---[ end trace 694817acca794f2c ]--- [60257.728500] ------------[ cut here ]------------ [60257.728500] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8() [60257.728500] Hardware name: [60257.728500] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys [60257.728500] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1 [60257.728500] Call Trace: [60257.728500] <IRQ> [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60257.728500] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60257.728500] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3 [60257.728500] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60257.728500] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d [60257.728500] [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net] [60257.728500] [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net] [60257.728500] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205 [60257.728500] [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole] [60257.728500] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f [60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60257.728500] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba [60257.728500] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b [60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60257.728500] [<ffffffff81308309>] ? printk+0x4e/0x5d [60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60257.728500] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf [60257.728500] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49 [60257.728500] [<ffffffff810626b7>] ? ktime_get+0xc/0x41 [60257.728500] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146 [60257.728500] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93 [60257.728500] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20 [60257.728500] <EOI> [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3 [60257.728500] [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31 [60257.728500] [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec [60257.728500] [<ffffffff81308f73>] ? thread_return+0x47/0xd5 [60257.728500] [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a [60257.728500] [<ffffffff81309213>] ? schedule_timeout+0x21/0x197 [60257.728500] [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10 [60257.728500] [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29 [60257.728500] [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3 [60257.728500] [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca [60257.728500] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e [60257.728500] [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac [60257.728500] [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd [60257.728500] [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5 [60257.728500] [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45 [60257.728500] [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d [60257.728500] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e [60257.728500] [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41 [60257.728500] [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd] [60257.728500] [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276 [60257.728500] [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd] [60257.728500] [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd] [60257.728500] [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd] [60257.728500] [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd] [60257.728500] [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd] [60257.728500] [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd] [60257.728500] [<ffffffff81011b9a>] ? child_rip+0xa/0x20 [60257.728500] [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4 [60257.728500] [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd] [60257.728500] [<ffffffff81011b90>] ? child_rip+0x0/0x20 [60257.728500] ---[ end trace 694817acca794f2d ]--- [60257.728500] ------------[ cut here ]------------ [60257.728500] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8() [60257.728500] Hardware name: [60257.728500] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys [60257.728500] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1 [60257.728500] Call Trace: [60257.728500] <IRQ> [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60257.728500] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60257.728500] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3 [60257.728500] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60257.728500] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d [60257.728500] [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net] [60257.728500] [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net] [60257.728500] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205 [60257.728500] [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole] [60257.728500] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f [60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60257.728500] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba [60257.728500] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b [60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60257.728500] [<ffffffff81308309>] ? printk+0x4e/0x5d [60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60257.728500] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf [60257.728500] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49 [60257.728500] [<ffffffff810626b7>] ? ktime_get+0xc/0x41 [60257.728500] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146 [60257.728500] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93 [60257.728500] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20 [60257.728500] <EOI> [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3 [60257.728500] [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31 [60257.728500] [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec [60257.728500] [<ffffffff81308f73>] ? thread_return+0x47/0xd5 [60257.728500] [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a [60257.728500] [<ffffffff81309213>] ? schedule_timeout+0x21/0x197 [60257.728500] [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10 [60257.728500] [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29 [60257.728500] [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3 [60257.728500] [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca [60257.728500] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e [60257.728500] [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac [60257.728500] [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd [60257.728500] [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5 [60257.728500] [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45 [60257.728500] [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d [60257.728500] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e [60257.728500] [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41 [60257.728500] [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd] [60257.728500] [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276 [60257.728500] [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd] [60257.728500] [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd] [60257.728500] [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd] [60257.728500] [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd] [60257.728500] [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd] [60257.728500] [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd] [60257.728500] [<ffffffff81011b9a>] ? child_rip+0xa/0x20 [60257.728500] [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4 [60257.728500] [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd] [60257.728500] [<ffffffff81011b90>] ? child_rip+0x0/0x20 [60257.728500] ---[ end trace 694817acca794f2e ]--- [60257.728500] ------------[ cut here ]------------ [60257.728500] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8() [60257.728500] Hardware name: [60257.728500] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys [60257.728500] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1 [60257.728500] Call Trace: [60257.728500] <IRQ> [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60257.728500] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60257.728500] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3 [60257.728500] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60257.728500] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d [60257.728500] [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net] [60257.728500] [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net] [60257.728500] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205 [60257.728500] [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole] [60257.728500] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f [60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60257.728500] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba [60257.728500] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b [60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60257.728500] [<ffffffff81308309>] ? printk+0x4e/0x5d [60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60257.728500] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf [60257.728500] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49 [60257.728500] [<ffffffff810626b7>] ? ktime_get+0xc/0x41 [60257.728500] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146 [60257.728500] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93 [60257.728500] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20 [60257.728500] <EOI> [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3 [60257.728500] [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31 [60257.728500] [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec [60257.728500] [<ffffffff81308f73>] ? thread_return+0x47/0xd5 [60257.728500] [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a [60257.728500] [<ffffffff81309213>] ? schedule_timeout+0x21/0x197 [60257.728500] [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10 [60257.728500] [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29 [60257.728500] [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3 [60257.728500] [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca [60257.728500] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e [60257.728500] [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac [60257.728500] [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd [60257.728500] [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5 [60257.728500] [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45 [60257.728500] [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d [60257.728500] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e [60257.728500] [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41 [60257.728500] [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd] [60257.728500] [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276 [60257.728500] [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd] [60257.728500] [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd] [60257.728500] [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd] [60257.728500] [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd] [60257.728500] [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd] [60257.728500] [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd] [60257.728500] [<ffffffff81011b9a>] ? child_rip+0xa/0x20 [60257.728500] [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4 [60257.728500] [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd] [60257.728500] [<ffffffff81011b90>] ? child_rip+0x0/0x20 [60257.728500] ---[ end trace 694817acca794f2f ]--- [60257.728500] ------------[ cut here ]------------ [60257.728500] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8() [60257.728500] Hardware name: [60257.728500] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys [60257.728500] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1 [60257.728500] Call Trace: [60257.728500] <IRQ> [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60257.728500] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60257.728500] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3 [60257.728500] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60257.728500] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d [60257.728500] [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net] [60257.728500] [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net] [60257.728500] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205 [60257.728500] [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole] [60257.728500] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f [60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60257.728500] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba [60257.728500] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b [60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60257.728500] [<ffffffff81308309>] ? printk+0x4e/0x5d [60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60257.728500] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf [60257.728500] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49 [60257.728500] [<ffffffff810626b7>] ? ktime_get+0xc/0x41 [60257.728500] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146 [60257.728500] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93 [60257.728500] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20 [60257.728500] <EOI> [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3 [60257.728500] [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31 [60257.728500] [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec [60257.728500] [<ffffffff81308f73>] ? thread_return+0x47/0xd5 [60257.728500] [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a [60257.728500] [<ffffffff81309213>] ? schedule_timeout+0x21/0x197 [60257.728500] [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10 [60257.728500] [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29 [60257.728500] [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3 [60257.728500] [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca [60257.728500] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e [60257.728500] [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac [60257.728500] [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd [60257.728500] [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5 [60257.728500] [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45 [60257.728500] [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d [60257.728500] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e [60257.728500] [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41 [60257.728500] [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd] [60257.728500] [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276 [60257.728500] [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd] [60257.728500] [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd] [60257.728500] [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd] [60257.728500] [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd] [60257.728500] [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd] [60257.728500] [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd] [60257.728500] [<ffffffff81011b9a>] ? child_rip+0xa/0x20 [60257.728500] [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4 [60257.728500] [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd] [60257.728500] [<ffffffff81011b90>] ? child_rip+0x0/0x20 [60257.728500] ---[ end trace 694817acca794f30 ]--- [60257.728500] ------------[ cut here ]------------ [60257.728500] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8() [60257.728500] Hardware name: [60257.728500] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys [60257.728500] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1 [60257.728500] Call Trace: [60257.728500] <IRQ> [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60257.728500] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60257.728500] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3 [60257.728500] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60257.728500] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d [60257.728500] [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net] [60257.728500] [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net] [60257.728500] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205 [60257.728500] [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole] [60257.728500] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f [60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60257.728500] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba [60257.728500] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b [60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60257.728500] [<ffffffff81308309>] ? printk+0x4e/0x5d [60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60257.728500] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf [60257.728500] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49 [60257.728500] [<ffffffff810626b7>] ? ktime_get+0xc/0x41 [60257.728500] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146 [60257.728500] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93 [60257.728500] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20 [60257.728500] <EOI> [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3 [60257.728500] [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31 [60257.728500] [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec [60257.728500] [<ffffffff81308f73>] ? thread_return+0x47/0xd5 [60257.728500] [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a [60257.728500] [<ffffffff81309213>] ? schedule_timeout+0x21/0x197 [60257.728500] [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10 [60257.728500] [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29 [60257.728500] [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3 [60257.728500] [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca [60257.728500] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e [60257.728500] [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac [60257.728500] [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd [60257.728500] [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5 [60257.728500] [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45 [60257.728500] [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d [60257.728500] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e [60257.728500] [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41 [60257.728500] [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd] [60257.728500] [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276 [60257.728500] [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd] [60257.728500] [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd] [60257.728500] [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd] [60257.728500] [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd] [60257.728500] [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd] [60257.728500] [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd] [60257.728500] [<ffffffff81011b9a>] ? child_rip+0xa/0x20 [60257.728500] [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4 [60257.728500] [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd] [60257.728500] [<ffffffff81011b90>] ? child_rip+0x0/0x20 [60257.728500] ---[ end trace 694817acca794f31 ]--- [60257.728500] ------------[ cut here ]------------ [60257.728500] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8() [60257.728500] Hardware name: [60257.728500] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys [60257.728500] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1 [60257.728500] Call Trace: [60257.728500] <IRQ> [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60257.728500] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60257.728500] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3 [60257.728500] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60257.728500] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d [60257.728500] [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net] [60257.728500] [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net] [60257.728500] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205 [60257.728500] [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole] [60257.728500] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f [60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60257.728500] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba [60257.728500] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b [60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60257.728500] [<ffffffff81308309>] ? printk+0x4e/0x5d [60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60257.728500] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf [60257.728500] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49 [60257.728500] [<ffffffff810626b7>] ? ktime_get+0xc/0x41 [60257.728500] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146 [60257.728500] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93 [60257.728500] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20 [60257.728500] <EOI> [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3 [60257.728500] [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31 [60257.728500] [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec [60257.728500] [<ffffffff81308f73>] ? thread_return+0x47/0xd5 [60257.728500] [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a [60257.728500] [<ffffffff81309213>] ? schedule_timeout+0x21/0x197 [60257.728500] [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10 [60257.728500] [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29 [60257.728500] [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3 [60257.728500] [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca [60257.728500] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e [60257.728500] [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac [60257.728500] [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd [60257.728500] [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5 [60257.728500] [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45 [60257.728500] [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d [60257.728500] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e [60257.728500] [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41 [60257.728500] [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd] [60257.728500] [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276 [60257.728500] [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd] [60257.728500] [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd] [60257.728500] [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd] [60257.728500] [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd] [60257.728500] [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd] [60257.728500] [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd] [60257.728500] [<ffffffff81011b9a>] ? child_rip+0xa/0x20 [60257.728500] [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4 [60257.728500] [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd] [60257.728500] [<ffffffff81011b90>] ? child_rip+0x0/0x20 [60257.728500] ---[ end trace 694817acca794f32 ]--- [60257.728500] ------------[ cut here ]------------ [60257.728500] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8() [60257.728500] Hardware name: [60257.728500] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys [60257.728500] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1 [60257.728500] Call Trace: [60257.728500] <IRQ> [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60257.728500] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60257.728500] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3 [60257.728500] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60257.728500] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d [60257.728500] [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net] [60257.728500] [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net] [60257.728500] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205 [60257.728500] [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole] [60257.728500] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f [60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60257.728500] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba [60257.728500] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b [60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60257.728500] [<ffffffff81308309>] ? printk+0x4e/0x5d [60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60257.728500] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf [60257.728500] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49 [60257.728500] [<ffffffff810626b7>] ? ktime_get+0xc/0x41 [60257.728500] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146 [60257.728500] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93 [60257.728500] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20 [60257.728500] <EOI> [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3 [60257.728500] [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31 [60257.728500] [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec [60257.728500] [<ffffffff81308f73>] ? thread_return+0x47/0xd5 [60257.728500] [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a [60257.728500] [<ffffffff81309213>] ? schedule_timeout+0x21/0x197 [60257.728500] [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10 [60257.728500] [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29 [60257.728500] [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3 [60257.728500] [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca [60257.728500] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e [60257.728500] [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac [60257.728500] [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd [60257.728500] [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5 [60257.728500] [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45 [60257.728500] [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d [60257.728500] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e [60257.728500] [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41 [60257.728500] [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd] [60257.728500] [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276 [60257.728500] [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd] [60257.728500] [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd] [60257.728500] [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd] [60257.728500] [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd] [60257.728500] [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd] [60257.728500] [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd] [60257.728500] [<ffffffff81011b9a>] ? child_rip+0xa/0x20 [60257.728500] [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4 [60257.728500] [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd] [60257.728500] [<ffffffff81011b90>] ? child_rip+0x0/0x20 [60257.728500] ---[ end trace 694817acca794f33 ]--- [60257.728500] ------------[ cut here ]------------ [60257.728500] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8() [60257.728500] Hardware name: [60257.728500] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys [60257.728500] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1 [60257.728500] Call Trace: [60257.728500] <IRQ> [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60257.728500] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60257.728500] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3 [60257.728500] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60257.728500] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d [60257.728500] [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net] [60257.728500] [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net] [60257.728500] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205 [60257.728500] [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole] [60257.728500] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f [60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60257.728500] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba [60257.728500] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b [60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60257.728500] [<ffffffff81308309>] ? printk+0x4e/0x5d [60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60257.728500] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf [60257.728500] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49 [60257.728500] [<ffffffff810626b7>] ? ktime_get+0xc/0x41 [60257.728500] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146 [60257.728500] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93 [60257.728500] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20 [60257.728500] <EOI> [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3 [60257.728500] [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31 [60257.728500] [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec [60257.728500] [<ffffffff81308f73>] ? thread_return+0x47/0xd5 [60257.728500] [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a [60257.728500] [<ffffffff81309213>] ? schedule_timeout+0x21/0x197 [60257.728500] [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10 [60257.728500] [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29 [60257.728500] [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3 [60257.728500] [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca [60257.728500] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e [60257.728500] [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac [60257.728500] [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd [60257.728500] [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5 [60257.728500] [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45 [60257.728500] [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d [60257.728500] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e [60257.728500] [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41 [60257.728500] [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd] [60257.728500] [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276 [60257.728500] [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd] [60257.728500] [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd] [60257.728500] [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd] [60257.728500] [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd] [60257.728500] [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd] [60257.728500] [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd] [60257.728500] [<ffffffff81011b9a>] ? child_rip+0xa/0x20 [60257.728500] [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4 [60257.728500] [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd] [60257.728500] [<ffffffff81011b90>] ? child_rip+0x0/0x20 [60257.728500] ---[ end trace 694817acca794f34 ]--- [60257.728500] ------------[ cut here ]------------ [60257.728500] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8() [60257.728500] Hardware name: [60257.728500] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys [60257.728500] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1 [60257.728500] Call Trace: [60257.728500] <IRQ> [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60257.728500] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60257.728500] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3 [60257.728500] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60257.728500] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d [60257.728500] [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net] [60257.728500] [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net] [60257.728500] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205 [60257.728500] [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole] [60257.728500] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f [60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60257.728500] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba [60257.728500] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b [60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60257.728500] [<ffffffff81308309>] ? printk+0x4e/0x5d [60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60257.728500] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf [60257.728500] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49 [60257.728500] [<ffffffff810626b7>] ? ktime_get+0xc/0x41 [60257.728500] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146 [60257.728500] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93 [60257.728500] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20 [60257.728500] <EOI> [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3 [60257.728500] [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31 [60257.728500] [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec [60257.728500] [<ffffffff81308f73>] ? thread_return+0x47/0xd5 [60257.728500] [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a [60257.728500] [<ffffffff81309213>] ? schedule_timeout+0x21/0x197 [60257.728500] [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10 [60257.728500] [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29 [60257.728500] [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3 [60257.728500] [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca [60257.728500] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e [60257.728500] [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac [60257.728500] [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd [60257.728500] [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5 [60257.728500] [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45 [60257.728500] [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d [60257.728500] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e [60257.728500] [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41 [60257.728500] [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd] [60257.728500] [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276 [60257.728500] [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd] [60257.728500] [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd] [60257.728500] [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd] [60257.728500] [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd] [60257.728500] [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd] [60257.728500] [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd] [60257.728500] [<ffffffff81011b9a>] ? child_rip+0xa/0x20 [60257.728500] [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4 [60257.728500] [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd] [60257.728500] [<ffffffff81011b90>] ? child_rip+0x0/0x20 [60257.728500] ---[ end trace 694817acca794f35 ]--- [60258.036152] ------------[ cut here ]------------ [60258.036152] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8() [60258.036152] Hardware name: [60258.036152] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys [60258.036152] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1 [60258.036152] Call Trace: [60258.036152] <IRQ> [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60258.036152] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60258.036152] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3 [60258.036152] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60258.036152] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d [60258.036152] [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net] [60258.036152] [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net] [60258.036152] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205 [60258.036152] [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole] [60258.036152] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f [60258.036152] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60258.036152] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba [60258.036152] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b [60258.036152] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60258.036152] [<ffffffff81308309>] ? printk+0x4e/0x5d [60258.036152] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60258.036152] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf [60258.036152] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49 [60258.036152] [<ffffffff810626b7>] ? ktime_get+0xc/0x41 [60258.036152] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146 [60258.036152] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93 [60258.036152] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20 [60258.036152] <EOI> [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3 [60258.036152] [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31 [60258.036152] [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec [60258.036152] [<ffffffff81308f73>] ? thread_return+0x47/0xd5 [60258.036152] [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a [60258.036152] [<ffffffff81309213>] ? schedule_timeout+0x21/0x197 [60258.036152] [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10 [60258.036152] [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29 [60258.036152] [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3 [60258.036152] [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca [60258.036152] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e [60258.036152] [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac [60258.036152] [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd [60258.036152] [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5 [60258.036152] [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45 [60258.036152] [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d [60258.036152] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e [60258.036152] [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41 [60258.036152] [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd] [60258.036152] [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276 [60258.036152] [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd] [60258.036152] [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd] [60258.036152] [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd] [60258.036152] [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd] [60258.036152] [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd] [60258.036152] [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd] [60258.036152] [<ffffffff81011b9a>] ? child_rip+0xa/0x20 [60258.036152] [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4 [60258.036152] [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd] [60258.036152] [<ffffffff81011b90>] ? child_rip+0x0/0x20 [60258.036152] ---[ end trace 694817acca794f36 ]--- [60258.036152] ------------[ cut here ]------------ [60258.036152] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8() [60258.036152] Hardware name: [60258.036152] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys [60258.036152] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1 [60258.036152] Call Trace: [60258.036152] <IRQ> [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60258.036152] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60258.036152] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3 [60258.036152] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60258.036152] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d [60258.036152] [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net] [60258.036152] [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net] [60258.036152] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205 [60258.036152] [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole] [60258.036152] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f [60258.036152] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60258.036152] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba [60258.036152] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b [60258.036152] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60258.036152] [<ffffffff81308309>] ? printk+0x4e/0x5d [60258.036152] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60258.036152] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf [60258.036152] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49 [60258.036152] [<ffffffff810626b7>] ? ktime_get+0xc/0x41 [60258.036152] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146 [60258.036152] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93 [60258.036152] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20 [60258.036152] <EOI> [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3 [60258.036152] [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31 [60258.036152] [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec [60258.036152] [<ffffffff81308f73>] ? thread_return+0x47/0xd5 [60258.036152] [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a [60258.036152] [<ffffffff81309213>] ? schedule_timeout+0x21/0x197 [60258.036152] [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10 [60258.036152] [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29 [60258.036152] [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3 [60258.036152] [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca [60258.036152] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e [60258.036152] [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac [60258.036152] [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd [60258.036152] [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5 [60258.036152] [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45 [60258.036152] [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d [60258.036152] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e [60258.036152] [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41 [60258.036152] [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd] [60258.036152] [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276 [60258.036152] [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd] [60258.036152] [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd] [60258.036152] [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd] [60258.036152] [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd] [60258.036152] [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd] [60258.036152] [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd] [60258.036152] [<ffffffff81011b9a>] ? child_rip+0xa/0x20 [60258.036152] [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4 [60258.036152] [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd] [60258.036152] [<ffffffff81011b90>] ? child_rip+0x0/0x20 [60258.036152] ---[ end trace 694817acca794f37 ]--- [60258.036152] ------------[ cut here ]------------ [60258.036152] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8() [60258.036152] Hardware name: [60258.036152] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys [60258.036152] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1 [60258.036152] Call Trace: [60258.036152] <IRQ> [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60258.036152] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60258.036152] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3 [60258.036152] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60258.036152] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d [60258.036152] [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net] [60258.036152] [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net] [60258.036152] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205 [60258.036152] [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole] [60258.036152] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f [60258.036152] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60258.036152] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba [60258.036152] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b [60258.036152] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60258.036152] [<ffffffff81308309>] ? printk+0x4e/0x5d [60258.036152] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60258.036152] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf [60258.036152] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49 [60258.036152] [<ffffffff810626b7>] ? ktime_get+0xc/0x41 [60258.036152] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146 [60258.036152] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93 [60258.036152] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20 [60258.036152] <EOI> [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3 [60258.036152] [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31 [60258.036152] [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec [60258.036152] [<ffffffff81308f73>] ? thread_return+0x47/0xd5 [60258.036152] [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a [60258.036152] [<ffffffff81309213>] ? schedule_timeout+0x21/0x197 [60258.036152] [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10 [60258.036152] [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29 [60258.036152] [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3 [60258.036152] [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca [60258.036152] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e [60258.036152] [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac [60258.036152] [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd [60258.036152] [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5 [60258.036152] [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45 [60258.036152] [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d [60258.036152] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e [60258.036152] [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41 [60258.036152] [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd] [60258.036152] [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276 [60258.036152] [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd] [60258.036152] [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd] [60258.036152] [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd] [60258.036152] [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd] [60258.036152] [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd] [60258.036152] [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd] [60258.036152] [<ffffffff81011b9a>] ? child_rip+0xa/0x20 [60258.036152] [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4 [60258.036152] [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd] [60258.036152] [<ffffffff81011b90>] ? child_rip+0x0/0x20 [60258.036152] ---[ end trace 694817acca794f38 ]--- [60258.036152] ------------[ cut here ]------------ [60258.036152] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8() [60258.036152] Hardware name: [60258.036152] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys [60258.036152] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1 [60258.036152] Call Trace: [60258.036152] <IRQ> [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60258.036152] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60258.036152] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3 [60258.036152] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60258.036152] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d [60258.036152] [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net] [60258.036152] [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net] [60258.036152] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205 [60258.036152] [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole] [60258.036152] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f [60258.036152] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60258.036152] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba [60258.036152] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b [60258.036152] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60258.036152] [<ffffffff81308309>] ? printk+0x4e/0x5d [60258.036152] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60258.036152] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf [60258.036152] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49 [60258.036152] [<ffffffff810626b7>] ? ktime_get+0xc/0x41 [60258.036152] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146 [60258.036152] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93 [60258.036152] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20 [60258.036152] <EOI> [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3 [60258.036152] [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31 [60258.036152] [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec [60258.036152] [<ffffffff81308f73>] ? thread_return+0x47/0xd5 [60258.036152] [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a [60258.036152] [<ffffffff81309213>] ? schedule_timeout+0x21/0x197 [60258.036152] [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10 [60258.036152] [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29 [60258.036152] [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3 [60258.036152] [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca [60258.036152] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e [60258.036152] [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac [60258.036152] [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd [60258.036152] [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5 [60258.036152] [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45 [60258.036152] [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d [60258.036152] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e [60258.036152] [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41 [60258.036152] [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd] [60258.036152] [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276 [60258.036152] [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd] [60258.036152] [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd] [60258.036152] [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd] [60258.036152] [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd] [60258.036152] [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd] [60258.036152] [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd] [60258.036152] [<ffffffff81011b9a>] ? child_rip+0xa/0x20 [60258.036152] [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4 [60258.036152] [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd] [60258.036152] [<ffffffff81011b90>] ? child_rip+0x0/0x20 [60258.036152] ---[ end trace 694817acca794f39 ]--- [60258.036152] ------------[ cut here ]------------ [60258.036152] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8() [60258.036152] Hardware name: [60258.036152] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys [60258.036152] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1 [60258.036152] Call Trace: [60258.036152] <IRQ> [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60258.036152] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60258.036152] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3 [60258.036152] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60258.036152] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d [60258.036152] [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net] [60258.036152] [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net] [60258.036152] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205 [60258.036152] [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole] [60258.036152] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f [60258.036152] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60258.036152] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba [60258.036152] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b [60258.036152] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60258.036152] [<ffffffff81308309>] ? printk+0x4e/0x5d [60258.036152] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60258.036152] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf [60258.036152] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49 [60258.036152] [<ffffffff810626b7>] ? ktime_get+0xc/0x41 [60258.036152] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146 [60258.036152] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93 [60258.036152] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20 [60258.036152] <EOI> [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3 [60258.036152] [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31 [60258.036152] [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec [60258.036152] [<ffffffff81308f73>] ? thread_return+0x47/0xd5 [60258.036152] [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a [60258.036152] [<ffffffff81309213>] ? schedule_timeout+0x21/0x197 [60258.036152] [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10 [60258.036152] [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29 [60258.036152] [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3 [60258.036152] [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca [60258.036152] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e [60258.036152] [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac [60258.036152] [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd [60258.036152] [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5 [60258.036152] [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45 [60258.036152] [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d [60258.036152] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e [60258.036152] [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41 [60258.036152] [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd] [60258.036152] [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276 [60258.036152] [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd] [60258.036152] [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd] [60258.036152] [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd] [60258.036152] [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd] [60258.036152] [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd] [60258.036152] [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd] [60258.036152] [<ffffffff81011b9a>] ? child_rip+0xa/0x20 [60258.036152] [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4 [60258.036152] [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd] [60258.036152] [<ffffffff81011b90>] ? child_rip+0x0/0x20 [60258.036152] ---[ end trace 694817acca794f3a ]--- [60258.036152] ------------[ cut here ]------------ [60258.036152] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8() [60258.036152] Hardware name: [60258.036152] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys [60258.036152] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1 [60258.036152] Call Trace: [60258.036152] <IRQ> [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60258.036152] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60258.036152] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3 [60258.036152] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60258.036152] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d [60258.036152] [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net] [60258.036152] [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net] [60258.036152] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205 [60258.036152] [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole] [60258.036152] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f [60258.036152] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60258.036152] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba [60258.036152] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b [60258.036152] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60258.036152] [<ffffffff81308309>] ? printk+0x4e/0x5d [60258.036152] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60258.036152] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf [60258.036152] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49 [60258.036152] [<ffffffff810626b7>] ? ktime_get+0xc/0x41 [60258.036152] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146 [60258.036152] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93 [60258.036152] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20 [60258.036152] <EOI> [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3 [60258.036152] [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31 [60258.036152] [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec [60258.036152] [<ffffffff81308f73>] ? thread_return+0x47/0xd5 [60258.036152] [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a [60258.036152] [<ffffffff81309213>] ? schedule_timeout+0x21/0x197 [60258.036152] [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10 [60258.036152] [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29 [60258.036152] [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3 [60258.036152] [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca [60258.036152] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e [60258.036152] [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac [60258.036152] [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd [60258.036152] [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5 [60258.036152] [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45 [60258.036152] [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d [60258.036152] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e [60258.036152] [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41 [60258.036152] [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd] [60258.036152] [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276 [60258.036152] [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd] [60258.036152] [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd] [60258.036152] [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd] [60258.036152] [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd] [60258.036152] [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd] [60258.036152] [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd] [60258.036152] [<ffffffff81011b9a>] ? child_rip+0xa/0x20 [60258.036152] [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4 [60258.036152] [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd] [60258.036152] [<ffffffff81011b90>] ? child_rip+0x0/0x20 [60258.036152] ---[ end trace 694817acca794f3b ]--- [60258.036152] ------------[ cut here ]------------ [60258.036152] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8() [60258.036152] Hardware name: [60258.036152] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys [60258.036152] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1 [60258.036152] Call Trace: [60258.036152] <IRQ> [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60258.036152] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60258.036152] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3 [60258.036152] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60258.036152] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d [60258.036152] [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net] [60258.036152] [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net] [60258.036152] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205 [60258.036152] [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole] [60258.036152] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f [60258.036152] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60258.036152] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba [60258.036152] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b [60258.036152] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60258.036152] [<ffffffff81308309>] ? printk+0x4e/0x5d [60258.036152] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60258.036152] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf [60258.036152] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49 [60258.036152] [<ffffffff810626b7>] ? ktime_get+0xc/0x41 [60258.036152] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146 [60258.036152] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93 [60258.036152] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20 [60258.036152] <EOI> [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3 [60258.036152] [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31 [60258.036152] [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec [60258.036152] [<ffffffff81308f73>] ? thread_return+0x47/0xd5 [60258.036152] [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a [60258.036152] [<ffffffff81309213>] ? schedule_timeout+0x21/0x197 [60258.036152] [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10 [60258.036152] [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29 [60258.036152] [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3 [60258.036152] [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca [60258.036152] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e [60258.036152] [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac [60258.036152] [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd [60258.036152] [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5 [60258.036152] [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45 [60258.036152] [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d [60258.036152] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e [60258.036152] [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41 [60258.036152] [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd] [60258.036152] [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276 [60258.036152] [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd] [60258.036152] [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd] [60258.036152] [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd] [60258.036152] [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd] [60258.036152] [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd] [60258.036152] [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd] [60258.036152] [<ffffffff81011b9a>] ? child_rip+0xa/0x20 [60258.036152] [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4 [60258.036152] [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd] [60258.036152] [<ffffffff81011b90>] ? child_rip+0x0/0x20 [60258.036152] ---[ end trace 694817acca794f3c ]--- [60259.116162] ------------[ cut here ]------------ [60259.116162] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8() [60259.116162] Hardware name: [60259.116162] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys [60259.116162] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1 [60259.116162] Call Trace: [60259.116162] <IRQ> [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60259.116162] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60259.116162] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3 [60259.116162] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60259.116162] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d [60259.116162] [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net] [60259.116162] [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net] [60259.116162] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205 [60259.116162] [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole] [60259.116162] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f [60259.116162] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60259.116162] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba [60259.116162] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b [60259.116162] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60259.116162] [<ffffffff81308309>] ? printk+0x4e/0x5d [60259.116162] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60259.116162] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf [60259.116162] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49 [60259.116162] [<ffffffff810626b7>] ? ktime_get+0xc/0x41 [60259.116162] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146 [60259.116162] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93 [60259.116162] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20 [60259.116162] <EOI> [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3 [60259.116162] [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31 [60259.116162] [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec [60259.116162] [<ffffffff81308f73>] ? thread_return+0x47/0xd5 [60259.116162] [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a [60259.116162] [<ffffffff81309213>] ? schedule_timeout+0x21/0x197 [60259.116162] [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10 [60259.116162] [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29 [60259.116162] [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3 [60259.116162] [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca [60259.116162] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e [60259.116162] [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac [60259.116162] [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd [60259.116162] [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5 [60259.116162] [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45 [60259.116162] [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d [60259.116162] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e [60259.116162] [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41 [60259.116162] [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd] [60259.116162] [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276 [60259.116162] [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd] [60259.116162] [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd] [60259.116162] [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd] [60259.116162] [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd] [60259.116162] [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd] [60259.116162] [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd] [60259.116162] [<ffffffff81011b9a>] ? child_rip+0xa/0x20 [60259.116162] [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4 [60259.116162] [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd] [60259.116162] [<ffffffff81011b90>] ? child_rip+0x0/0x20 [60259.116162] ---[ end trace 694817acca794f3d ]--- [60259.116162] ------------[ cut here ]------------ [60259.116162] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8() [60259.116162] Hardware name: [60259.116162] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys [60259.116162] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1 [60259.116162] Call Trace: [60259.116162] <IRQ> [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60259.116162] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60259.116162] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3 [60259.116162] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60259.116162] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d [60259.116162] [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net] [60259.116162] [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net] [60259.116162] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205 [60259.116162] [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole] [60259.116162] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f [60259.116162] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60259.116162] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba [60259.116162] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b [60259.116162] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60259.116162] [<ffffffff81308309>] ? printk+0x4e/0x5d [60259.116162] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60259.116162] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf [60259.116162] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49 [60259.116162] [<ffffffff810626b7>] ? ktime_get+0xc/0x41 [60259.116162] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146 [60259.116162] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93 [60259.116162] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20 [60259.116162] <EOI> [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3 [60259.116162] [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31 [60259.116162] [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec [60259.116162] [<ffffffff81308f73>] ? thread_return+0x47/0xd5 [60259.116162] [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a [60259.116162] [<ffffffff81309213>] ? schedule_timeout+0x21/0x197 [60259.116162] [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10 [60259.116162] [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29 [60259.116162] [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3 [60259.116162] [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca [60259.116162] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e [60259.116162] [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac [60259.116162] [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd [60259.116162] [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5 [60259.116162] [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45 [60259.116162] [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d [60259.116162] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e [60259.116162] [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41 [60259.116162] [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd] [60259.116162] [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276 [60259.116162] [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd] [60259.116162] [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd] [60259.116162] [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd] [60259.116162] [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd] [60259.116162] [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd] [60259.116162] [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd] [60259.116162] [<ffffffff81011b9a>] ? child_rip+0xa/0x20 [60259.116162] [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4 [60259.116162] [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd] [60259.116162] [<ffffffff81011b90>] ? child_rip+0x0/0x20 [60259.116162] ---[ end trace 694817acca794f3e ]--- [60259.116162] ------------[ cut here ]------------ [60259.116162] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8() [60259.116162] Hardware name: [60259.116162] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys [60259.116162] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1 [60259.116162] Call Trace: [60259.116162] <IRQ> [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60259.116162] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60259.116162] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3 [60259.116162] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60259.116162] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d [60259.116162] [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net] [60259.116162] [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net] [60259.116162] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205 [60259.116162] [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole] [60259.116162] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f [60259.116162] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60259.116162] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba [60259.116162] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b [60259.116162] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60259.116162] [<ffffffff81308309>] ? printk+0x4e/0x5d [60259.116162] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60259.116162] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf [60259.116162] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49 [60259.116162] [<ffffffff810626b7>] ? ktime_get+0xc/0x41 [60259.116162] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146 [60259.116162] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93 [60259.116162] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20 [60259.116162] <EOI> [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3 [60259.116162] [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31 [60259.116162] [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec [60259.116162] [<ffffffff81308f73>] ? thread_return+0x47/0xd5 [60259.116162] [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a [60259.116162] [<ffffffff81309213>] ? schedule_timeout+0x21/0x197 [60259.116162] [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10 [60259.116162] [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29 [60259.116162] [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3 [60259.116162] [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca [60259.116162] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e [60259.116162] [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac [60259.116162] [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd [60259.116162] [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5 [60259.116162] [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45 [60259.116162] [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d [60259.116162] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e [60259.116162] [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41 [60259.116162] [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd] [60259.116162] [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276 [60259.116162] [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd] [60259.116162] [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd] [60259.116162] [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd] [60259.116162] [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd] [60259.116162] [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd] [60259.116162] [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd] [60259.116162] [<ffffffff81011b9a>] ? child_rip+0xa/0x20 [60259.116162] [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4 [60259.116162] [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd] [60259.116162] [<ffffffff81011b90>] ? child_rip+0x0/0x20 [60259.116162] ---[ end trace 694817acca794f3f ]--- [60259.116162] ------------[ cut here ]------------ [60259.116162] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8() [60259.116162] Hardware name: [60259.116162] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys [60259.116162] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1 [60259.116162] Call Trace: [60259.116162] <IRQ> [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60259.116162] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60259.116162] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3 [60259.116162] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60259.116162] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d [60259.116162] [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net] [60259.116162] [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net] [60259.116162] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205 [60259.116162] [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole] [60259.116162] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f [60259.116162] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60259.116162] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba [60259.116162] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b [60259.116162] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60259.116162] [<ffffffff81308309>] ? printk+0x4e/0x5d [60259.116162] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60259.116162] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf [60259.116162] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49 [60259.116162] [<ffffffff810626b7>] ? ktime_get+0xc/0x41 [60259.116162] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146 [60259.116162] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93 [60259.116162] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20 [60259.116162] <EOI> [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3 [60259.116162] [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31 [60259.116162] [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec [60259.116162] [<ffffffff81308f73>] ? thread_return+0x47/0xd5 [60259.116162] [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a [60259.116162] [<ffffffff81309213>] ? schedule_timeout+0x21/0x197 [60259.116162] [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10 [60259.116162] [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29 [60259.116162] [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3 [60259.116162] [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca [60259.116162] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e [60259.116162] [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac [60259.116162] [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd [60259.116162] [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5 [60259.116162] [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45 [60259.116162] [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d [60259.116162] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e [60259.116162] [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41 [60259.116162] [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd] [60259.116162] [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276 [60259.116162] [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd] [60259.116162] [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd] [60259.116162] [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd] [60259.116162] [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd] [60259.116162] [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd] [60259.116162] [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd] [60259.116162] [<ffffffff81011b9a>] ? child_rip+0xa/0x20 [60259.116162] [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4 [60259.116162] [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd] [60259.116162] [<ffffffff81011b90>] ? child_rip+0x0/0x20 [60259.116162] ---[ end trace 694817acca794f40 ]--- [60259.116162] ------------[ cut here ]------------ [60259.116162] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8() [60259.116162] Hardware name: [60259.116162] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys [60259.116162] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1 [60259.116162] Call Trace: [60259.116162] <IRQ> [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60259.116162] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60259.116162] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3 [60259.116162] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60259.116162] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d [60259.116162] [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net] [60259.116162] [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net] [60259.116162] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205 [60259.116162] [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole] [60259.116162] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f [60259.116162] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60259.116162] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba [60259.116162] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b [60259.116162] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60259.116162] [<ffffffff81308309>] ? printk+0x4e/0x5d [60259.116162] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60259.116162] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf [60259.116162] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49 [60259.116162] [<ffffffff810626b7>] ? ktime_get+0xc/0x41 [60259.116162] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146 [60259.116162] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93 [60259.116162] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20 [60259.116162] <EOI> [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3 [60259.116162] [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31 [60259.116162] [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec [60259.116162] [<ffffffff81308f73>] ? thread_return+0x47/0xd5 [60259.116162] [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a [60259.116162] [<ffffffff81309213>] ? schedule_timeout+0x21/0x197 [60259.116162] [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10 [60259.116162] [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29 [60259.116162] [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3 [60259.116162] [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca [60259.116162] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e [60259.116162] [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac [60259.116162] [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd [60259.116162] [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5 [60259.116162] [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45 [60259.116162] [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d [60259.116162] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e [60259.116162] [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41 [60259.116162] [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd] [60259.116162] [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276 [60259.116162] [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd] [60259.116162] [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd] [60259.116162] [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd] [60259.116162] [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd] [60259.116162] [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd] [60259.116162] [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd] [60259.116162] [<ffffffff81011b9a>] ? child_rip+0xa/0x20 [60259.116162] [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4 [60259.116162] [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd] [60259.116162] [<ffffffff81011b90>] ? child_rip+0x0/0x20 [60259.116162] ---[ end trace 694817acca794f41 ]--- [60259.116162] ------------[ cut here ]------------ [60259.116162] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8() [60259.116162] Hardware name: [60259.116162] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys [60259.116162] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1 [60259.116162] Call Trace: [60259.116162] <IRQ> [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60259.116162] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60259.116162] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3 [60259.116162] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60259.116162] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d [60259.116162] [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net] [60259.116162] [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net] [60259.116162] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205 [60259.116162] [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole] [60259.116162] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f [60259.116162] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60259.116162] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba [60259.116162] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b [60259.116162] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60259.116162] [<ffffffff81308309>] ? printk+0x4e/0x5d [60259.116162] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60259.116162] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf [60259.116162] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49 [60259.116162] [<ffffffff810626b7>] ? ktime_get+0xc/0x41 [60259.116162] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146 [60259.116162] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93 [60259.116162] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20 [60259.116162] <EOI> [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3 [60259.116162] [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31 [60259.116162] [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec [60259.116162] [<ffffffff81308f73>] ? thread_return+0x47/0xd5 [60259.116162] [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a [60259.116162] [<ffffffff81309213>] ? schedule_timeout+0x21/0x197 [60259.116162] [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10 [60259.116162] [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29 [60259.116162] [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3 [60259.116162] [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca [60259.116162] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e [60259.116162] [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac [60259.116162] [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd [60259.116162] [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5 [60259.116162] [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45 [60259.116162] [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d [60259.116162] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e [60259.116162] [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41 [60259.116162] [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd] [60259.116162] [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276 [60259.116162] [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd] [60259.116162] [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd] [60259.116162] [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd] [60259.116162] [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd] [60259.116162] [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd] [60259.116162] [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd] [60259.116162] [<ffffffff81011b9a>] ? child_rip+0xa/0x20 [60259.116162] [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4 [60259.116162] [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd] [60259.116162] [<ffffffff81011b90>] ? child_rip+0x0/0x20 [60259.116162] ---[ end trace 694817acca794f42 ]--- [60259.116162] ------------[ cut here ]------------ [60259.116162] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8() [60259.116162] Hardware name: [60259.116162] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys [60259.116162] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1 [60259.116162] Call Trace: [60259.116162] <IRQ> [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60259.116162] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60259.116162] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3 [60259.116162] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60259.116162] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d [60259.116162] [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net] [60259.116162] [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net] [60259.116162] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205 [60259.116162] [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole] [60259.116162] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f [60259.116162] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60259.116162] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba [60259.116162] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b [60259.116162] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60259.116162] [<ffffffff81308309>] ? printk+0x4e/0x5d [60259.116162] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60259.116162] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf [60259.116162] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49 [60259.116162] [<ffffffff810626b7>] ? ktime_get+0xc/0x41 [60259.116162] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146 [60259.116162] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93 [60259.116162] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20 [60259.116162] <EOI> [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3 [60259.116162] [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31 [60259.116162] [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec [60259.116162] [<ffffffff81308f73>] ? thread_return+0x47/0xd5 [60259.116162] [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a [60259.116162] [<ffffffff81309213>] ? schedule_timeout+0x21/0x197 [60259.116162] [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10 [60259.116162] [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29 [60259.116162] [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3 [60259.116162] [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca [60259.116162] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e [60259.116162] [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac [60259.116162] [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd [60259.116162] [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5 [60259.116162] [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45 [60259.116162] [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d [60259.116162] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e [60259.116162] [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41 [60259.116162] [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd] [60259.116162] [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276 [60259.116162] [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd] [60259.116162] [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd] [60259.116162] [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd] [60259.116162] [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd] [60259.116162] [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd] [60259.116162] [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd] [60259.116162] [<ffffffff81011b9a>] ? child_rip+0xa/0x20 [60259.116162] [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4 [60259.116162] [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd] [60259.116162] [<ffffffff81011b90>] ? child_rip+0x0/0x20 [60259.116162] ---[ end trace 694817acca794f43 ]--- [60259.116162] ------------[ cut here ]------------ [60259.116162] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8() [60259.116162] Hardware name: [60259.116162] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys [60259.116162] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1 [60259.116162] Call Trace: [60259.116162] <IRQ> [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60259.116162] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60259.116162] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3 [60259.116162] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60259.116162] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d [60259.116162] [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net] [60259.116162] [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net] [60259.116162] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205 [60259.116162] [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole] [60259.116162] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f [60259.116162] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba [60259.116162] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b [60259.116162] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60259.116162] [<ffffffff81308309>] ? printk+0x4e/0x5d [60259.116162] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60259.116162] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf [60259.116162] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49 [60259.116162] [<ffffffff810626b7>] ? ktime_get+0xc/0x41 [60259.116162] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146 [60259.116162] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93 [60259.116162] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20 [60259.116162] <EOI> [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3 [60259.116162] [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31 [60259.116162] [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec [60259.116162] [<ffffffff81308f73>] ? thread_return+0x47/0xd5 [60259.116162] [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a [60259.116162] [<ffffffff81309213>] ? schedule_timeout+0x21/0x197 [60259.116162] [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10 [60259.116162] [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29 [60259.116162] [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3 [60259.116162] [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca [60259.116162] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e [60259.116162] [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac [60259.116162] [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd [60259.116162] [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5 [60259.116162] [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45 [60259.116162] [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d [60259.116162] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e [60259.116162] [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41 [60259.116162] [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd] [60259.116162] [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276 [60259.116162] [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd] [60259.116162] [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd] [60259.116162] [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd] [60259.116162] [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd] [60259.116162] [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd] [60259.116162] [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd] [60259.116162] [<ffffffff81011b9a>] ? child_rip+0xa/0x20 [60259.116162] [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4 [60259.116162] [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd] [60259.116162] [<ffffffff81011b90>] ? child_rip+0x0/0x20 [60259.116162] ---[ end trace 694817acca794f44 ]--- [60259.116162] ------------[ cut here ]------------ [60259.116162] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8() [60259.116162] Hardware name: [60259.116162] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys [60259.116162] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1 [60259.116162] Call Trace: [60259.116162] <IRQ> [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60259.116162] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60259.116162] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3 [60259.116162] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60259.116162] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d [60259.116162] [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net] [60259.116162] [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net] [60259.116162] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205 [60259.116162] [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole] [60259.116162] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f [60259.116162] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba [60259.116162] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b [60259.116162] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60259.116162] [<ffffffff81308309>] ? printk+0x4e/0x5d [60259.116162] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60259.116162] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf [60259.116162] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49 [60259.116162] [<ffffffff810626b7>] ? ktime_get+0xc/0x41 [60259.116162] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146 [60259.116162] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93 [60259.116162] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20 [60259.116162] <EOI> [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3 [60259.116162] [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31 [60259.116162] [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec [60259.116162] [<ffffffff81308f73>] ? thread_return+0x47/0xd5 [60259.116162] [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a [60259.116162] [<ffffffff81309213>] ? schedule_timeout+0x21/0x197 [60259.116162] [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10 [60259.116162] [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29 [60259.116162] [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3 [60259.116162] [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca [60259.116162] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e [60259.116162] [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac [60259.116162] [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd [60259.116162] [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5 [60259.116162] [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45 [60259.116162] [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d [60259.116162] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e [60259.116162] [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41 [60259.116162] [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd] [60259.116162] [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276 [60259.116162] [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd] [60259.116162] [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd] [60259.116162] [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd] [60259.116162] [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd] [60259.116162] [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd] [60259.116162] [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd] [60259.116162] [<ffffffff81011b9a>] ? child_rip+0xa/0x20 [60259.116162] [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4 [60259.116162] [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd] [60259.116162] [<ffffffff81011b90>] ? child_rip+0x0/0x20 [60259.116162] ---[ end trace 694817acca794f45 ]--- [60259.116162] ------------[ cut here ]------------ [60259.116162] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8() [60259.116162] Hardware name: [60259.116162] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys [60259.116162] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1 [60259.116162] Call Trace: [60259.116162] <IRQ> [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60259.116162] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60259.116162] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3 [60259.116162] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60259.116162] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d [60259.116162] [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net] [60259.116162] [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net] [60259.116162] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205 [60259.116162] [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole] [60259.116162] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f [60259.116162] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba [60259.116162] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b [60259.116162] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60259.116162] [<ffffffff81308309>] ? printk+0x4e/0x5d [60259.116162] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60259.116162] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf [60259.116162] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49 [60259.116162] [<ffffffff810626b7>] ? ktime_get+0xc/0x41 [60259.116162] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146 [60259.116162] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93 [60259.116162] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20 [60259.116162] <EOI> [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3 [60259.116162] [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31 [60259.116162] [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec [60259.116162] [<ffffffff81308f73>] ? thread_return+0x47/0xd5 [60259.116162] [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a [60259.116162] [<ffffffff81309213>] ? schedule_timeout+0x21/0x197 [60259.116162] [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10 [60259.116162] [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29 [60259.116162] [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3 [60259.116162] [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca [60259.116162] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e [60259.116162] [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac [60259.116162] [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd [60259.116162] [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5 [60259.116162] [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45 [60259.116162] [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d [60259.116162] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e [60259.116162] [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41 [60259.116162] [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd] [60259.116162] [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276 [60259.116162] [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd] [60259.116162] [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd] [60259.116162] [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd] [60259.116162] [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd] [60259.116162] [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd] [60259.116162] [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd] [60259.116162] [<ffffffff81011b9a>] ? child_rip+0xa/0x20 [60259.116162] [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4 [60259.116162] [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd] [60259.116162] [<ffffffff81011b90>] ? child_rip+0x0/0x20 [60259.116162] ---[ end trace 694817acca794f46 ]--- [60259.116162] ------------[ cut here ]------------ [60259.116162] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8() [60259.116162] Hardware name: [60259.116162] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys [60259.116162] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1 [60259.116162] Call Trace: [60259.116162] <IRQ> [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60259.116162] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60259.116162] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3 [60259.116162] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 [60259.116162] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d [60259.116162] [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net] [60259.116162] [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net] [60259.116162] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205 [60259.116162] [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole] [60259.116162] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f [60259.116162] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba [60259.116162] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b [60259.116162] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60259.116162] [<ffffffff81308309>] ? printk+0x4e/0x5d [60259.116162] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 [60259.116162] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf [60259.116162] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49 [60259.116162] [<ffffffff810626b7>] ? ktime_get+0xc/0x41 [60259.116162] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146 [60259.116162] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93 [60259.116162] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20 [60259.116162] <EOI> [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3 [60259.116162] [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31 [60259.116162] [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec [60259.116162] [<ffffffff81308f73>] ? thread_return+0x47/0xd5 [60259.116162] [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a [60259.116162] [<ffffffff81309213>] ? schedule_timeout+0x21/0x197 [60259.116162] [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10 [60259.116162] [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29 [60259.116162] [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3 [60259.116162] [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca [60259.116162] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e [60259.116162] [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac [60259.116162] [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd [60259.116162] [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5 [60259.116162] [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45 [60259.116162] [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d [60259.116162] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e [60259.116162] [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41 [60259.116162] [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd] [60259.116162] [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276 [60259.116162] [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd] [60259.116162] [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd] [60259.116162] [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd] [60259.116162] [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd] [60259.116162] [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd] [60259.116162] [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd] [60259.116162] [<ffffffff81011b9a>] ? child_rip+0xa/0x20 [60259.116162] [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4 [60259.116162] [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd] [60259.116162] [<ffffffff81011b90>] ? child_rip+0x0/0x20 [60259.116162] ---[ end trace 694817acca794f47 ]--- ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Bug #14378] Problems with net/core/skbuff.c 2009-10-13 9:11 ` [Bug #14378] Problems with net/core/skbuff.c Massimo Cetra @ 2009-10-13 9:23 ` Massimo Cetra 2009-10-13 10:19 ` Eric Dumazet [not found] ` <4AD44435.3050703-BBpJ+9iBSNKonA0d6jMUrA@public.gmane.org> 2 siblings, 0 replies; 12+ messages in thread From: Massimo Cetra @ 2009-10-13 9:23 UTC (permalink / raw) To: Massimo Cetra; +Cc: David Miller, rjw, linux-kernel, kernel-testers, netdev Massimo Cetra ha scritto: > David Miller ha scritto: >> From: "Rafael J. Wysocki" <rjw@sisk.pl> >> Date: Mon, 12 Oct 2009 00:22:04 +0200 (CEST) >> >> >>> Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14378 >>> Subject : Problems with net/core/skbuff.c >>> Submitter : Massimo Cetra <mcetra@navynet.it> >>> Date : 2009-10-08 14:51 (4 days old) >>> References : http://marc.info/?l=linux-kernel&m=125501488220358&w=4 >>> >> >> I don't know what to do about this one. >> >> The user indicates that they have the vserver patches applied, >> so maybe there is some interaction with that stuff. >> > Actually i found another oops which is very similar to the previous one. And here it is another one, this time triggered by postfix, where mor drbd nor vserver are involved. This is not the same server where the other oopses were grabbed. Max ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Bug #14378] Problems with net/core/skbuff.c 2009-10-13 9:11 ` [Bug #14378] Problems with net/core/skbuff.c Massimo Cetra 2009-10-13 9:23 ` Massimo Cetra @ 2009-10-13 10:19 ` Eric Dumazet 2009-10-13 10:25 ` David Miller 2009-10-13 10:26 ` Massimo Cetra [not found] ` <4AD44435.3050703-BBpJ+9iBSNKonA0d6jMUrA@public.gmane.org> 2 siblings, 2 replies; 12+ messages in thread From: Eric Dumazet @ 2009-10-13 10:19 UTC (permalink / raw) To: Massimo Cetra; +Cc: David Miller, rjw, linux-kernel, kernel-testers, netdev Massimo Cetra a écrit : > David Miller ha scritto: >> From: "Rafael J. Wysocki" <rjw@sisk.pl> >> Date: Mon, 12 Oct 2009 00:22:04 +0200 (CEST) >> >> >>> Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14378 >>> Subject : Problems with net/core/skbuff.c >>> Submitter : Massimo Cetra <mcetra@navynet.it> >>> Date : 2009-10-08 14:51 (4 days old) >>> References : http://marc.info/?l=linux-kernel&m=125501488220358&w=4 >>> >> >> I don't know what to do about this one. >> >> The user indicates that they have the vserver patches applied, >> so maybe there is some interaction with that stuff. >> > Actually i found another oops which is very similar to the previous one. > Here, vserver is not involved, and the problem starts at drbd which > lives in kernel space (the other oops started at ocfs2). > > Both ocfs2 and drbd make heavy use of network I/O so i guess the problem > is something in the network layer. > > Anything i can do to help to debugging and solving this issue ? > > Thanks > Max > Problem is kfree_skb() is called from irq context, wich is not allowed. static void skb_release_head_state(struct sk_buff *skb) { ... if (skb->destructor) { WARN_ON(in_irq()); skb->destructor(); } ... } virtio_net start_xmit() function calls free_old_xmit_skbs() and free_old_xmit_skbs() ultimately calls kfree_skb() Quick fix would be to use dev_kfree_skb_any() instead, because netpoll can definitly calls start_xmit() with irq disabled. Could you please following patch ? diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c index 8d00976..54bf091 100644 --- a/drivers/net/virtio_net.c +++ b/drivers/net/virtio_net.c @@ -454,7 +454,7 @@ static unsigned int free_old_xmit_skbs(struct virtnet_info *vi) vi->dev->stats.tx_bytes += skb->len; vi->dev->stats.tx_packets++; tot_sgs += skb_vnet_hdr(skb)->num_sg; - kfree_skb(skb); + dev_kfree_skb_any(skb); } return tot_sgs; } ^ permalink raw reply related [flat|nested] 12+ messages in thread
* Re: [Bug #14378] Problems with net/core/skbuff.c 2009-10-13 10:19 ` Eric Dumazet @ 2009-10-13 10:25 ` David Miller 2009-10-13 10:26 ` Massimo Cetra 1 sibling, 0 replies; 12+ messages in thread From: David Miller @ 2009-10-13 10:25 UTC (permalink / raw) To: eric.dumazet; +Cc: ctrixk, rjw, linux-kernel, kernel-testers, netdev From: Eric Dumazet <eric.dumazet@gmail.com> Date: Tue, 13 Oct 2009 12:19:51 +0200 > Quick fix would be to use dev_kfree_skb_any() instead, > because netpoll can definitly calls start_xmit() > with irq disabled. Indeed, because of netpoll(), that is something drivers must be able to cope with. > Could you please following patch ? Indeed, let us know how Eric's patch works. Thanks Eric! ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Bug #14378] Problems with net/core/skbuff.c 2009-10-13 10:19 ` Eric Dumazet 2009-10-13 10:25 ` David Miller @ 2009-10-13 10:26 ` Massimo Cetra 2009-10-14 19:27 ` Massimo Cetra 1 sibling, 1 reply; 12+ messages in thread From: Massimo Cetra @ 2009-10-13 10:26 UTC (permalink / raw) To: Eric Dumazet Cc: Massimo Cetra, David Miller, rjw, linux-kernel, kernel-testers, netdev Eric Dumazet ha scritto: > Could you please following patch ? > > diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c > index 8d00976..54bf091 100644 > --- a/drivers/net/virtio_net.c > +++ b/drivers/net/virtio_net.c > @@ -454,7 +454,7 @@ static unsigned int free_old_xmit_skbs(struct virtnet_info *vi) > vi->dev->stats.tx_bytes += skb->len; > vi->dev->stats.tx_packets++; > tot_sgs += skb_vnet_hdr(skb)->num_sg; > - kfree_skb(skb); > + dev_kfree_skb_any(skb); > } > return tot_sgs; > } > > > Thank you very much. Compiling. It' will be in production in a few minutes. I'll let you know if the problem arises again. give me a couple of days because this problem is randomly triggered. Sometimes it happens multiple times a day, sometimes none. Max ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Bug #14378] Problems with net/core/skbuff.c 2009-10-13 10:26 ` Massimo Cetra @ 2009-10-14 19:27 ` Massimo Cetra [not found] ` <4AD62626.6010709-BBpJ+9iBSNKonA0d6jMUrA@public.gmane.org> 0 siblings, 1 reply; 12+ messages in thread From: Massimo Cetra @ 2009-10-14 19:27 UTC (permalink / raw) To: Eric Dumazet Cc: Massimo Cetra, David Miller, rjw, linux-kernel, kernel-testers, netdev Massimo Cetra ha scritto: > Eric Dumazet ha scritto: >> Could you please following patch ? >> >> diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c >> index 8d00976..54bf091 100644 >> --- a/drivers/net/virtio_net.c >> +++ b/drivers/net/virtio_net.c >> @@ -454,7 +454,7 @@ static unsigned int free_old_xmit_skbs(struct >> virtnet_info *vi) >> vi->dev->stats.tx_bytes += skb->len; >> vi->dev->stats.tx_packets++; >> tot_sgs += skb_vnet_hdr(skb)->num_sg; >> - kfree_skb(skb); >> + dev_kfree_skb_any(skb); >> } >> return tot_sgs; >> } >> >> >> > Thank you very much. > > Compiling. > It' will be in production in a few minutes. > I'll let you know if the problem arises again. > give me a couple of days because this problem is randomly triggered. > Sometimes it happens multiple times a day, sometimes none. > Eric, thanks for the patch. The problem didn't arise again and i haven't seen any warning like that on both servers where that problem was happening more frequently. I would say that it's fixed and if it's not, i'll let you know as soon as it happens again. Thanks again, Max ^ permalink raw reply [flat|nested] 12+ messages in thread
[parent not found: <4AD62626.6010709-BBpJ+9iBSNKonA0d6jMUrA@public.gmane.org>]
* [PATCH] virtio_net: use dev_kfree_skb_any() in free_old_xmit_skbs() [not found] ` <4AD62626.6010709-BBpJ+9iBSNKonA0d6jMUrA@public.gmane.org> @ 2009-10-15 0:36 ` Eric Dumazet 2009-10-15 6:30 ` David Miller 0 siblings, 1 reply; 12+ messages in thread From: Eric Dumazet @ 2009-10-15 0:36 UTC (permalink / raw) To: Massimo Cetra Cc: Massimo Cetra, David Miller, rjw-KKrjLPT3xs0, linux-kernel-u79uwXL29TY76Z2rM5mHXA, kernel-testers-u79uwXL29TY76Z2rM5mHXA, netdev-u79uwXL29TY76Z2rM5mHXA Massimo Cetra a écrit : > Eric, > thanks for the patch. > The problem didn't arise again and i haven't seen any warning like that > on both servers where that problem was happening more frequently. > > I would say that it's fixed and if it's not, i'll let you know as soon > as it happens again. > Thanks Massimo, I think patch is reasonably safe and should be taken as is : [PATCH] virtio_net: use dev_kfree_skb_any() in free_old_xmit_skbs() Because netpoll can call netdevice start_xmit() method with irqs disabled, drivers should not call kfree_skb() from their start_xmit(), but use dev_kfree_skb_any() instead. Oct 8 11:16:52 172.30.1.31 [113074.791813] ------------[ cut here ]------------ Oct 8 11:16:52 172.30.1.31 [113074.791813] WARNING: at net/core/skbuff.c:398 \ skb_release_head_state+0x64/0xc8() Oct 8 11:16:52 172.30.1.31 [113074.791813] Hardware name: Oct 8 11:16:52 172.30.1.31 [113074.791813] Modules linked in: netconsole ocfs2 jbd2 quota_tree \ ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue configfs crc32c drbd cn loop \ serio_raw psmouse snd_pcm snd_timer snd soundcore snd_page_alloc virtio_net pcspkr parport_pc parport \ i2c_piix4 i2c_core button processor evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot \ dm_mod ide_cd_mod cdrom ata_generic ata_piix virtio_blk libata scsi_mod piix ide_pci_generic ide_core \ virtio_pci virtio_ring virtio floppy thermal fan thermal_sys [last unloaded: netconsole] Oct 8 11:16:52 172.30.1.31 [113074.791813] Pid: 11132, comm: php5-cgi Tainted: G W \ 2.6.31.2-vserver #1 Oct 8 11:16:52 172.30.1.31 [113074.791813] Call Trace: Oct 8 11:16:52 172.30.1.31 [113074.791813] <IRQ> [<ffffffff81253cd5>] ? \ skb_release_head_state+0x64/0xc8 Oct 8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 Oct 8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3 Oct 8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8 Oct 8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d Oct 8 11:16:52 172.30.1.31 [113074.791813] [<ffffffffa01cb139>] ? free_old_xmit_skbs+0x51/0x6e \ [virtio_net] Oct 8 11:16:52 172.30.1.31 [113074.791813] [<ffffffffa01cbc85>] ? start_xmit+0x26/0xf2 [virtio_net] Oct 8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205 Oct 8 11:16:52 172.30.1.31 [113074.791813] [<ffffffffa0429216>] ? write_msg+0x90/0xeb [netconsole] Oct 8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f Oct 8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 Oct 8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba Oct 8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b Oct 8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff8106b142>] ? vx_update_load+0x18/0x13e Oct 8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff81308309>] ? printk+0x4e/0x5d Oct 8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52 Oct 8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf Oct 8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49 Oct 8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff810626b7>] ? ktime_get+0xc/0x41 Oct 8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146 Oct 8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93 Oct 8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20 Oct 8 11:16:52 172.30.1.31 [113074.791813] <EOI> [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31 Reported-and-tested-by: Massimo Cetra <mcetra-BBpJ+9iBSNKonA0d6jMUrA@public.gmane.org> Signed-off-by: Eric Dumazet <eric.dumazet-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> Bug-Entry: http://bugzilla.kernel.org/show_bug.cgi?id=14378 --- diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c index 8d00976..54bf091 100644 --- a/drivers/net/virtio_net.c +++ b/drivers/net/virtio_net.c @@ -454,7 +454,7 @@ static unsigned int free_old_xmit_skbs(struct virtnet_info *vi) vi->dev->stats.tx_bytes += skb->len; vi->dev->stats.tx_packets++; tot_sgs += skb_vnet_hdr(skb)->num_sg; - kfree_skb(skb); + dev_kfree_skb_any(skb); } return tot_sgs; } ^ permalink raw reply related [flat|nested] 12+ messages in thread
* Re: [PATCH] virtio_net: use dev_kfree_skb_any() in free_old_xmit_skbs() 2009-10-15 0:36 ` [PATCH] virtio_net: use dev_kfree_skb_any() in free_old_xmit_skbs() Eric Dumazet @ 2009-10-15 6:30 ` David Miller 0 siblings, 0 replies; 12+ messages in thread From: David Miller @ 2009-10-15 6:30 UTC (permalink / raw) To: eric.dumazet; +Cc: mcetra, ctrixk, rjw, linux-kernel, kernel-testers, netdev From: Eric Dumazet <eric.dumazet@gmail.com> Date: Thu, 15 Oct 2009 02:36:43 +0200 > Massimo Cetra a écrit : >> Eric, >> thanks for the patch. >> The problem didn't arise again and i haven't seen any warning like that >> on both servers where that problem was happening more frequently. >> >> I would say that it's fixed and if it's not, i'll let you know as soon >> as it happens again. >> > > Thanks Massimo, I think patch is reasonably safe and should be taken as is : > > > [PATCH] virtio_net: use dev_kfree_skb_any() in free_old_xmit_skbs() Applied, thanks Eric. ^ permalink raw reply [flat|nested] 12+ messages in thread
[parent not found: <4AD44435.3050703-BBpJ+9iBSNKonA0d6jMUrA@public.gmane.org>]
* Re: [Bug #14378] Problems with net/core/skbuff.c [not found] ` <4AD44435.3050703-BBpJ+9iBSNKonA0d6jMUrA@public.gmane.org> @ 2009-10-13 10:22 ` David Miller [not found] ` <20091013.032237.45775304.davem-fT/PcQaiUtIeIZ0/mPfg9Q@public.gmane.org> 0 siblings, 1 reply; 12+ messages in thread From: David Miller @ 2009-10-13 10:22 UTC (permalink / raw) To: ctrixk-BBpJ+9iBSNKonA0d6jMUrA Cc: rjw-KKrjLPT3xs0, linux-kernel-u79uwXL29TY76Z2rM5mHXA, kernel-testers-u79uwXL29TY76Z2rM5mHXA, netdev-u79uwXL29TY76Z2rM5mHXA From: Massimo Cetra <ctrixk-BBpJ+9iBSNKonA0d6jMUrA@public.gmane.org> Date: Tue, 13 Oct 2009 11:11:17 +0200 > Here, vserver is not involved, and the problem starts at drbd which > lives in kernel space (the other oops started at ocfs2). > > Both ocfs2 and drbd make heavy use of network I/O so i guess the > problem is something in the network layer. > > Anything i can do to help to debugging and solving this issue ? Is all of your traffic going over virtio_net? ^ permalink raw reply [flat|nested] 12+ messages in thread
[parent not found: <20091013.032237.45775304.davem-fT/PcQaiUtIeIZ0/mPfg9Q@public.gmane.org>]
* Re: [Bug #14378] Problems with net/core/skbuff.c [not found] ` <20091013.032237.45775304.davem-fT/PcQaiUtIeIZ0/mPfg9Q@public.gmane.org> @ 2009-10-13 10:27 ` Massimo Cetra 0 siblings, 0 replies; 12+ messages in thread From: Massimo Cetra @ 2009-10-13 10:27 UTC (permalink / raw) To: David Miller Cc: ctrixk-BBpJ+9iBSNKonA0d6jMUrA, rjw-KKrjLPT3xs0, linux-kernel-u79uwXL29TY76Z2rM5mHXA, kernel-testers-u79uwXL29TY76Z2rM5mHXA, netdev-u79uwXL29TY76Z2rM5mHXA David Miller ha scritto: > From: Massimo Cetra <ctrixk-BBpJ+9iBSNKonA0d6jMUrA@public.gmane.org> > Date: Tue, 13 Oct 2009 11:11:17 +0200 > > >> Here, vserver is not involved, and the problem starts at drbd which >> lives in kernel space (the other oops started at ocfs2). >> >> Both ocfs2 and drbd make heavy use of network I/O so i guess the >> problem is something in the network layer. >> >> Anything i can do to help to debugging and solving this issue ? >> > > Is all of your traffic going over virtio_net? > Yes, indeed. I'm compiling the patch and i'll let you know. Thanks, Max ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2009-10-15 6:30 UTC | newest]
Thread overview: 12+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2009-10-11 22:07 2.6.32-rc4: Reported regressions from 2.6.31 Rafael J. Wysocki
[not found] ` <r-4Wjt4coy.A.TPD.dal0KB@chimera>
[not found] ` <b637ec0b0910121148g34f46481n1d5d4edb67b0ac55@mail.gmail.com>
2009-10-12 21:55 ` [Bug #14372] Wireless not working after suspend-resume Rafael J. Wysocki
[not found] ` <r-4Wjt4coy.A.CwC.Wal0KB@chimera>
[not found] ` <20091012.034224.64980795.davem@davemloft.net>
2009-10-13 9:11 ` [Bug #14378] Problems with net/core/skbuff.c Massimo Cetra
2009-10-13 9:23 ` Massimo Cetra
2009-10-13 10:19 ` Eric Dumazet
2009-10-13 10:25 ` David Miller
2009-10-13 10:26 ` Massimo Cetra
2009-10-14 19:27 ` Massimo Cetra
[not found] ` <4AD62626.6010709-BBpJ+9iBSNKonA0d6jMUrA@public.gmane.org>
2009-10-15 0:36 ` [PATCH] virtio_net: use dev_kfree_skb_any() in free_old_xmit_skbs() Eric Dumazet
2009-10-15 6:30 ` David Miller
[not found] ` <4AD44435.3050703-BBpJ+9iBSNKonA0d6jMUrA@public.gmane.org>
2009-10-13 10:22 ` [Bug #14378] Problems with net/core/skbuff.c David Miller
[not found] ` <20091013.032237.45775304.davem-fT/PcQaiUtIeIZ0/mPfg9Q@public.gmane.org>
2009-10-13 10:27 ` Massimo Cetra
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).