[Bug #14278] New message "NOHZ: local_softirq_pending 08" at each ping request

[Bug #14278] New message "NOHZ: local_softirq_pending 08" at each ping request

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14278
Subject		: New message "NOHZ: local_softirq_pending 08" at each ping request
Submitter	: Christian Casteyde <casteyde.christian-GANU6spQydw@public.gmane.org>
Date		: 2009-09-30 18:12 (2 days old)
Handled-By	: Michael Buesch <mb-fseUSCV1ubazQB+pC5nmwQ@public.gmane.org>
Patch		: http://bugzilla.kernel.org/attachment.cgi?id=23220

2.6.32-rc4: Reported regressions from 2.6.31

[Rafael J. Wysocki]

[Note:
  We started to get more regression reports for 2.6.32-rc (100%+ jump in the
  last 10 days.  Nevertheless, we're still receiving new reports of regressions
  from 2.6.30.]

This message contains a list of some regressions from 2.6.31, for which there
are no fixes in the mainline I know of.  If any of them have been fixed already,
please let me know.

If you know of any other unresolved regressions from 2.6.31, please let me know
either and I'll add them to the list.  Also, please let me know if any of the
entries below are invalid.

Each entry from the list will be sent additionally in an automatic reply to
this message with CCs to the people involved in reporting and handling the
issue.


Listed regressions statistics:

  Date          Total  Pending  Unresolved
  ----------------------------------------
  2009-10-12       48       31          27
  2009-10-02       22       15           9


Unresolved regressions
----------------------

Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14392
Subject		: Touchpad "paste" stops working after suspend to RAM
Submitter	: Carlos R. Mafra <crmafra2@gmail.com>
Date		: 2009-10-11 16:21 (1 days old)
References	: http://marc.info/?l=linux-kernel&m=125527987316493&w=4
Handled-By	: Dmitry Torokhov <dmitry.torokhov@gmail.com>


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14390
Subject		: "bind" a device to a driver doesn't not work anymore
Submitter	: Éric Piel <Eric.Piel@tremplin-utc.net>
Date		: 2009-10-11 0:04 (1 days old)
References	: http://marc.info/?l=linux-kernel&m=125521979921241&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14389
Subject		: Build system issue
Submitter	: Peter Zijlstra <peterz@infradead.org>
Date		: 2009-10-09 8:58 (3 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=575543347b5baed0ca927cb90ba8807396fe9cc9
References	: http://marc.info/?l=linux-kernel&m=125507914909152&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14387
Subject		: deadlock with fallocate
Submitter	: Thomas Neumann <tneumann@users.sourceforge.net>
Date		: 2009-10-07 3:00 (5 days old)
References	: http://marc.info/?l=linux-kernel&m=125488495526471&w=4
Handled-By	: Christoph Hellwig <hch@lst.de>


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14386
Subject		: GPF in snd_hda_intel
Submitter	: Luca Tettamanti <kronos.it@gmail.com>
Date		: 2009-10-10 13:01 (2 days old)
References	: http://marc.info/?l=linux-kernel&m=125517999408019&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14384
Subject		: tbench regression with 2.6.32-rc1
Submitter	: Zhang, Yanmin <yanmin_zhang@linux.intel.com>
Date		: 2009-10-09 9:51 (3 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=59abf02644c45f1591e1374ee7bb45dc757fcb88
References	: http://marc.info/?l=linux-kernel&m=125508216713138&w=4
Handled-By	: Peter Zijlstra <a.p.zijlstra@chello.nl>


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14383
Subject		: hackbench regression with kernel 2.6.32-rc1
Submitter	: Zhang, Yanmin <yanmin_zhang@linux.intel.com>
Date		: 2009-10-09 9:19 (3 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=29cd8bae396583a2ee9a3340db8c5102acf9f6fd
References	: http://marc.info/?l=linux-kernel&m=125508007510274&w=4
Handled-By	: Peter Zijlstra <a.p.zijlstra@chello.nl>


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14381
Subject		: iwlagn lost connection after s2ram (with warnings)
Submitter	: Carlos R. Mafra <crmafra2@gmail.com>
Date		: 2009-10-07 14:20 (5 days old)
References	: http://marc.info/?l=linux-kernel&m=125492569119947&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14380
Subject		: Video tearing/glitching with T400 laptops
Submitter	: Theodore Ts'o <tytso@mit.edu>
Date		: 2009-10-02 22:40 (10 days old)
References	: http://marc.info/?l=linux-kernel&m=125452324520623&w=4
Handled-By	: Jesse Barnes <jbarnes@virtuousgeek.org>


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14379
Subject		: ACPI Warning for _SB_.BAT0._BIF: Converted Buffer to expected String
Submitter	: Justin Mattock <justinmattock@gmail.com>
Date		: 2009-10-08 21:46 (4 days old)
References	: http://marc.info/?l=linux-kernel&m=125504031328941&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14378
Subject		: Problems with net/core/skbuff.c
Submitter	: Massimo Cetra <mcetra@navynet.it>
Date		: 2009-10-08 14:51 (4 days old)
References	: http://marc.info/?l=linux-kernel&m=125501488220358&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14376
Subject		: Kernel NULL pointer dereference/ kvm subsystem
Submitter	: Don Dupuis <dondster@gmail.com>
Date		: 2009-10-06 14:38 (6 days old)
References	: http://marc.info/?l=linux-kernel&m=125484025021737&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14374
Subject		: MCEs caused by commit db8be50c4307dac2b37305fc59c8dc0f978d09ea
Submitter	: Nick Piggin <npiggin@suse.de>
Date		: 2009-10-02 7:34 (10 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=db8be50c4307dac2b37305fc59c8dc0f978d09ea
References	: http://marc.info/?l=linux-kernel&m=125446885705223&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14373
Subject		: Task blocked for more than 120 seconds
Submitter	: Zeno Davatz <zdavatz@gmail.com>
Date		: 2009-10-02 10:16 (10 days old)
References	: http://marc.info/?l=linux-kernel&m=125447858618412&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14372
Subject		: Wireless not working after suspend-resume
Submitter	: Fabio Comolli <fabio.comolli@gmail.com>
Date		: 2009-10-03 15:36 (9 days old)
References	: http://lkml.org/lkml/2009/10/3/91


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14370
Subject		: ext4 corruptions
Submitter	: Alexey Fisher <bug-track@fisher-privat.net>
Date		: 2009-10-09 19:20 (3 days old)
References	: http://marc.info/?l=linux-kernel&m=125511643504864&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14355
Subject		: USB serial regression after 2.6.31.1 with Huawei E169 GSM modem
Submitter	: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Date		: 2009-10-10 03:07 (2 days old)
References	: http://marc.info/?l=linux-kernel&m=125513456327542&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14354
Subject		: Bad corruption with 2.6.32-rc1 and upwards
Submitter	: Holger Freyther <zecke@selfish.org>
Date		: 2009-10-09 15:42 (3 days old)


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14353
Subject		: BUG: sleeping function called from invalid context at kernel/mutex.c:280
Submitter	: Miles Lane <miles.lane@gmail.com>
Date		: 2009-10-05 3:39 (7 days old)
References	: http://marc.info/?l=linux-kernel&m=125471432208671&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14352
Subject		: WARNING: at net/mac80211/scan.c:267
Submitter	: Maciej Rutecki <maciej.rutecki@gmail.com>
Date		: 2009-10-08 00:30 (4 days old)
References	: http://bugzilla.intellinuxwireless.org/show_bug.cgi?id=2089#c7


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14334
Subject		: pcmcia suspend regression from 2.6.31.1 to 2.6.31.2 - Dell Inspiron 600m
Submitter	: Jose Marino <braket@hotmail.com>
Date		: 2009-10-06 15:44 (6 days old)


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14299
Subject		: oops in wireless, iwl3945 related?
Submitter	: Pavel Machek <pavel@ucw.cz>
Date		: 2009-09-29 17:12 (13 days old)
References	: http://marc.info/?l=linux-kernel&m=125424439725743&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14298
Subject		: warning at manage.c:361 (set_irq_wake), matrix-keypad related?
Submitter	: Pavel Machek <pavel@ucw.cz>
Date		: 2009-09-30 20:07 (12 days old)
References	: http://marc.info/?l=linux-kernel&m=125434130703538&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14297
Subject		: console resume broken since ba15ab0e8d
Submitter	: Sascha Hauer <s.hauer@pengutronix.de>
Date		: 2009-09-30 15:11 (12 days old)
References	: http://marc.info/?l=linux-kernel&m=125432349404060&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14296
Subject		: spitz boots but suspend/resume is broken
Submitter	: Pavel Machek <pavel@ucw.cz>
Date		: 2009-09-30 12:06 (12 days old)
References	: http://marc.info/?l=linux-kernel&m=125431244516449&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14279
Subject		: Suspend to RAM freeze totally since 2.6.32-rc1 - Acer Aspire 1511Lmi laptop
Submitter	: Christian Casteyde <casteyde.christian@free.fr>
Date		: 2009-09-30 18:14 (12 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=5f68563996e812f9ca35b3939ad2a42e5d254d66


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14277
Subject		: Caught 8-bit read from freed memory in b43 driver at association
Submitter	: Christian Casteyde <casteyde.christian@free.fr>
Date		: 2009-09-30 18:06 (12 days old)


Regressions with patches
------------------------

Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14382
Subject		: Transmit failure in et131x.
Submitter	: Nick Bowler <nbowler@elliptictech.com>
Date		: 2009-10-08 14:08 (4 days old)
References	: http://marc.info/?l=linux-kernel&m=125501117713744&w=4
Handled-By	: Alan Cox <alan@lxorguk.ukuu.org.uk>
Patch		: http://patchwork.kernel.org/patch/52698/


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14375
Subject		: Intel(R) I/OAT DMA Engine init failed
Submitter	: Alexander Beregalov <a.beregalov@gmail.com>
Date		: 2009-10-02 9:46 (10 days old)
References	: http://marc.info/?l=linux-kernel&m=125447680016160&w=4
Handled-By	: Dan Williams <dan.j.williams@intel.com>
Patch		: http://patchwork.kernel.org/patch/51808/


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14302
Subject		: Kernel panic on i386 machine when booting with profile=2
Submitter	: Shi, Alex <alex.shi@intel.com>
Date		: 2009-10-01 3:23 (11 days old)
References	: http://marc.info/?l=linux-kernel&m=125436749607199&w=4
Handled-By	: Alex Shi <alex.shi@intel.com>
Patch		: http://patchwork.kernel.org/patch/50813/


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14278
Subject		: New message "NOHZ: local_softirq_pending 08" at each ping request
Submitter	: Christian Casteyde <casteyde.christian@free.fr>
Date		: 2009-09-30 18:12 (12 days old)
Handled-By	: Michael Buesch <mb@bu3sch.de>
Patch		: http://bugzilla.kernel.org/attachment.cgi?id=23220


For details, please visit the bug entries and follow the links given in
references.

As you can see, there is a Bugzilla entry for each of the listed regressions.
There also is a Bugzilla entry used for tracking the regressions from 2.6.31,
unresolved as well as resolved, at:

http://bugzilla.kernel.org/show_bug.cgi?id=14230

Please let me know if there are any Bugzilla entries that should be added to
the list in there.

Thanks,
Rafael

_______________________________________________
linux-pm mailing list
linux-pm@lists.linux-foundation.org
https://lists.linux-foundation.org/mailman/listinfo/linux-pm

[Bug #14277] Caught 8-bit read from freed memory in b43 driver at association

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14277
Subject		: Caught 8-bit read from freed memory in b43 driver at association
Submitter	: Christian Casteyde <casteyde.christian-GANU6spQydw@public.gmane.org>
Date		: 2009-09-30 18:06 (12 days old)

[Bug #14278] New message "NOHZ: local_softirq_pending 08" at each ping request

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14278
Subject		: New message "NOHZ: local_softirq_pending 08" at each ping request
Submitter	: Christian Casteyde <casteyde.christian-GANU6spQydw@public.gmane.org>
Date		: 2009-09-30 18:12 (12 days old)
Handled-By	: Michael Buesch <mb-fseUSCV1ubazQB+pC5nmwQ@public.gmane.org>
Patch		: http://bugzilla.kernel.org/attachment.cgi?id=23220

[Bug #14296] spitz boots but suspend/resume is broken

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14296
Subject		: spitz boots but suspend/resume is broken
Submitter	: Pavel Machek <pavel@ucw.cz>
Date		: 2009-09-30 12:06 (12 days old)
References	: http://marc.info/?l=linux-kernel&m=125431244516449&w=4

[Bug #14279] Suspend to RAM freeze totally since 2.6.32-rc1 - Acer Aspire 1511Lmi laptop

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14279
Subject		: Suspend to RAM freeze totally since 2.6.32-rc1 - Acer Aspire 1511Lmi laptop
Submitter	: Christian Casteyde <casteyde.christian-GANU6spQydw@public.gmane.org>
Date		: 2009-09-30 18:14 (12 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=5f68563996e812f9ca35b3939ad2a42e5d254d66

[Bug #14297] console resume broken since ba15ab0e8d

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14297
Subject		: console resume broken since ba15ab0e8d
Submitter	: Sascha Hauer <s.hauer-bIcnvbaLZ9MEGnE8C9+IrQ@public.gmane.org>
Date		: 2009-09-30 15:11 (12 days old)
References	: http://marc.info/?l=linux-kernel&m=125432349404060&w=4

[Bug #14298] warning at manage.c:361 (set_irq_wake), matrix-keypad related?

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14298
Subject		: warning at manage.c:361 (set_irq_wake), matrix-keypad related?
Submitter	: Pavel Machek <pavel@ucw.cz>
Date		: 2009-09-30 20:07 (12 days old)
References	: http://marc.info/?l=linux-kernel&m=125434130703538&w=4

[Bug #14334] pcmcia suspend regression from 2.6.31.1 to 2.6.31.2 - Dell Inspiron 600m

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14334
Subject		: pcmcia suspend regression from 2.6.31.1 to 2.6.31.2 - Dell Inspiron 600m
Submitter	: Jose Marino <braket@hotmail.com>
Date		: 2009-10-06 15:44 (6 days old)

[Bug #14352] WARNING: at net/mac80211/scan.c:267

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14352
Subject		: WARNING: at net/mac80211/scan.c:267
Submitter	: Maciej Rutecki <maciej.rutecki-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
Date		: 2009-10-08 00:30 (4 days old)
References	: http://bugzilla.intellinuxwireless.org/show_bug.cgi?id=2089#c7

[Bug #14302] Kernel panic on i386 machine when booting with profile=2

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14302
Subject		: Kernel panic on i386 machine when booting with profile=2
Submitter	: Shi, Alex <alex.shi-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Date		: 2009-10-01 3:23 (11 days old)
References	: http://marc.info/?l=linux-kernel&m=125436749607199&w=4
Handled-By	: Alex Shi <alex.shi-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Patch		: http://patchwork.kernel.org/patch/50813/

[Bug #14299] oops in wireless, iwl3945 related?

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14299
Subject		: oops in wireless, iwl3945 related?
Submitter	: Pavel Machek <pavel@ucw.cz>
Date		: 2009-09-29 17:12 (13 days old)
References	: http://marc.info/?l=linux-kernel&m=125424439725743&w=4

[Bug #14353] BUG: sleeping function called from invalid context at kernel/mutex.c:280

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14353
Subject		: BUG: sleeping function called from invalid context at kernel/mutex.c:280
Submitter	: Miles Lane <miles.lane-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
Date		: 2009-10-05 3:39 (7 days old)
References	: http://marc.info/?l=linux-kernel&m=125471432208671&w=4

[Bug #14370] ext4 corruptions

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14370
Subject		: ext4 corruptions
Submitter	: Alexey Fisher <bug-track@fisher-privat.net>
Date		: 2009-10-09 19:20 (3 days old)
References	: http://marc.info/?l=linux-kernel&m=125511643504864&w=4

[Bug #14355] USB serial regression after 2.6.31.1 with Huawei E169 GSM modem

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14355
Subject		: USB serial regression after 2.6.31.1 with Huawei E169 GSM modem
Submitter	: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Date		: 2009-10-10 03:07 (2 days old)
References	: http://marc.info/?l=linux-kernel&m=125513456327542&w=4

[Bug #14354] Bad corruption with 2.6.32-rc1 and upwards

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14354
Subject		: Bad corruption with 2.6.32-rc1 and upwards
Submitter	: Holger Freyther <zecke@selfish.org>
Date		: 2009-10-09 15:42 (3 days old)

[Bug #14374] MCEs caused by commit db8be50c4307dac2b37305fc59c8dc0f978d09ea

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14374
Subject		: MCEs caused by commit db8be50c4307dac2b37305fc59c8dc0f978d09ea
Submitter	: Nick Piggin <npiggin@suse.de>
Date		: 2009-10-02 7:34 (10 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=db8be50c4307dac2b37305fc59c8dc0f978d09ea
References	: http://marc.info/?l=linux-kernel&m=125446885705223&w=4

[Bug #14373] Task blocked for more than 120 seconds

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14373
Subject		: Task blocked for more than 120 seconds
Submitter	: Zeno Davatz <zdavatz-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
Date		: 2009-10-02 10:16 (10 days old)
References	: http://marc.info/?l=linux-kernel&m=125447858618412&w=4

[Bug #14372] Wireless not working after suspend-resume

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14372
Subject		: Wireless not working after suspend-resume
Submitter	: Fabio Comolli <fabio.comolli-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
Date		: 2009-10-03 15:36 (9 days old)
References	: http://lkml.org/lkml/2009/10/3/91

[Bug #14376] Kernel NULL pointer dereference/ kvm subsystem

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14376
Subject		: Kernel NULL pointer dereference/ kvm subsystem
Submitter	: Don Dupuis <dondster-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
Date		: 2009-10-06 14:38 (6 days old)
References	: http://marc.info/?l=linux-kernel&m=125484025021737&w=4

[Bug #14375] Intel(R) I/OAT DMA Engine init failed

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14375
Subject		: Intel(R) I/OAT DMA Engine init failed
Submitter	: Alexander Beregalov <a.beregalov-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
Date		: 2009-10-02 9:46 (10 days old)
References	: http://marc.info/?l=linux-kernel&m=125447680016160&w=4
Handled-By	: Dan Williams <dan.j.williams-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Patch		: http://patchwork.kernel.org/patch/51808/

[Bug #14378] Problems with net/core/skbuff.c

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14378
Subject		: Problems with net/core/skbuff.c
Submitter	: Massimo Cetra <mcetra-BBpJ+9iBSNKonA0d6jMUrA@public.gmane.org>
Date		: 2009-10-08 14:51 (4 days old)
References	: http://marc.info/?l=linux-kernel&m=125501488220358&w=4

[Bug #14381] iwlagn lost connection after s2ram (with warnings)

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14381
Subject		: iwlagn lost connection after s2ram (with warnings)
Submitter	: Carlos R. Mafra <crmafra2@gmail.com>
Date		: 2009-10-07 14:20 (5 days old)
References	: http://marc.info/?l=linux-kernel&m=125492569119947&w=4

[Bug #14382] Transmit failure in et131x.

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14382
Subject		: Transmit failure in et131x.
Submitter	: Nick Bowler <nbowler-7BP4RkwGw0uXmMXjJBpWqg@public.gmane.org>
Date		: 2009-10-08 14:08 (4 days old)
References	: http://marc.info/?l=linux-kernel&m=125501117713744&w=4
Handled-By	: Alan Cox <alan-qBU/x9rampVanCEyBjwyrvXRex20P6io@public.gmane.org>
Patch		: http://patchwork.kernel.org/patch/52698/

[Bug #14379] ACPI Warning for _SB_.BAT0._BIF: Converted Buffer to expected String

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14379
Subject		: ACPI Warning for _SB_.BAT0._BIF: Converted Buffer to expected String
Submitter	: Justin Mattock <justinmattock-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
Date		: 2009-10-08 21:46 (4 days old)
References	: http://marc.info/?l=linux-kernel&m=125504031328941&w=4

[Bug #14380] Video tearing/glitching with T400 laptops

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14380
Subject		: Video tearing/glitching with T400 laptops
Submitter	: Theodore Ts'o <tytso-3s7WtUTddSA@public.gmane.org>
Date		: 2009-10-02 22:40 (10 days old)
References	: http://marc.info/?l=linux-kernel&m=125452324520623&w=4
Handled-By	: Jesse Barnes <jbarnes-Y1mF5jBUw70BENJcbMCuUQ@public.gmane.org>

[Bug #14384] tbench regression with 2.6.32-rc1

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14384
Subject		: tbench regression with 2.6.32-rc1
Submitter	: Zhang, Yanmin <yanmin_zhang@linux.intel.com>
Date		: 2009-10-09 9:51 (3 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=59abf02644c45f1591e1374ee7bb45dc757fcb88
References	: http://marc.info/?l=linux-kernel&m=125508216713138&w=4
Handled-By	: Peter Zijlstra <a.p.zijlstra@chello.nl>

[Bug #14386] GPF in snd_hda_intel

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14386
Subject		: GPF in snd_hda_intel
Submitter	: Luca Tettamanti <kronos.it-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
Date		: 2009-10-10 13:01 (2 days old)
References	: http://marc.info/?l=linux-kernel&m=125517999408019&w=4

[Bug #14383] hackbench regression with kernel 2.6.32-rc1

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14383
Subject		: hackbench regression with kernel 2.6.32-rc1
Submitter	: Zhang, Yanmin <yanmin_zhang-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
Date		: 2009-10-09 9:19 (3 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=29cd8bae396583a2ee9a3340db8c5102acf9f6fd
References	: http://marc.info/?l=linux-kernel&m=125508007510274&w=4
Handled-By	: Peter Zijlstra <a.p.zijlstra-/NLkJaSkS4VmR6Xm/wNWPw@public.gmane.org>

[Bug #14387] deadlock with fallocate

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14387
Subject		: deadlock with fallocate
Submitter	: Thomas Neumann <tneumann-Rn4VEauK+AKRv+LV9MX5uipxlwaOVQ5f@public.gmane.org>
Date		: 2009-10-07 3:00 (5 days old)
References	: http://marc.info/?l=linux-kernel&m=125488495526471&w=4
Handled-By	: Christoph Hellwig <hch-jcswGhMUV9g@public.gmane.org>

[Bug #14392] Touchpad "paste" stops working after suspend to RAM

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14392
Subject		: Touchpad "paste" stops working after suspend to RAM
Submitter	: Carlos R. Mafra <crmafra2-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
Date		: 2009-10-11 16:21 (1 days old)
References	: http://marc.info/?l=linux-kernel&m=125527987316493&w=4
Handled-By	: Dmitry Torokhov <dmitry.torokhov-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>

[Bug #14390] "bind" a device to a driver doesn't not work anymore

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14390
Subject		: "bind" a device to a driver doesn't not work anymore
Submitter	: Éric Piel <Eric.Piel-VkQ1JFuSMpfAbQlEx87xDw@public.gmane.org>
Date		: 2009-10-11 0:04 (1 days old)
References	: http://marc.info/?l=linux-kernel&m=125521979921241&w=4

[Bug #14389] Build system issue

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14389
Subject		: Build system issue
Submitter	: Peter Zijlstra <peterz-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>
Date		: 2009-10-09 8:58 (3 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=575543347b5baed0ca927cb90ba8807396fe9cc9
References	: http://marc.info/?l=linux-kernel&m=125507914909152&w=4

Re: [Bug #14382] Transmit failure in et131x.

[Alan Cox]

On Mon, 12 Oct 2009 00:22:05 +0200 (CEST)
"Rafael J. Wysocki" <rjw-KKrjLPT3xs0@public.gmane.org> wrote:

> This message has been generated automatically as a part of a report
> of recent regressions.
> 
> The following bug entry is on the current list of known regressions
> from 2.6.31.  Please verify if it still should be listed and let me know
> (either way).

Patch sent for testing, nothing heard back. I think I've fixed it - I
certainly found a bug 8)

Re: [Bug #14379] ACPI Warning for _SB_.BAT0._BIF: Converted Buffer to expected String

[Justin Mattock]

On Sun, Oct 11, 2009 at 3:22 PM, Rafael J. Wysocki <rjw-KKrjLPT3xs0@public.gmane.org> wrote:
> This message has been generated automatically as a part of a report
> of recent regressions.
>
> The following bug entry is on the current list of known regressions
> from 2.6.31.  Please verify if it still should be listed and let me know
> (either way).
>
>
> Bug-Entry       : http://bugzilla.kernel.org/show_bug.cgi?id=14379
> Subject         : ACPI Warning for _SB_.BAT0._BIF: Converted Buffer to expected String
> Submitter       : Justin Mattock <justinmattock-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
> Date            : 2009-10-08 21:46 (4 days old)
> References      : http://marc.info/?l=linux-kernel&m=125504031328941&w=4
>
>
>

pulled rc4 today and the warning is still present.
here is the results of git bisect:

d9adc2e031bd22d5d9607a53a8d3b30e0b675f39 is the first bad commit
commit d9adc2e031bd22d5d9607a53a8d3b30e0b675f39
Author: Lin Ming <ming.m.lin-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Date:   Mon Jul 27 11:31:10 2009 +0800

    ACPICA: reformat predefined method table, no functional change

    Reformatted the methods that return package objects.

    Signed-off-by: Lin Ming <ming.m.lin-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
    Signed-off-by: Len Brown <len.brown-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>

:040000 040000 17c30a6e75ce2fafc677b32bdcf3fbecae8458ab
f43e38a2a4034bcc73ef14943e61187f11b08731 M      drivers

I will go and attach the git log to the bug report.

-- 
Justin P. Mattock

Re: [Bug #14379] ACPI Warning for _SB_.BAT0._BIF: Converted Buffer to expected String

[Lin Ming]

On Mon, 2009-10-12 at 11:14 +0800, Justin Mattock wrote:
> On Sun, Oct 11, 2009 at 3:22 PM, Rafael J. Wysocki <rjw-KKrjLPT3xs0@public.gmane.org> wrote:
> > This message has been generated automatically as a part of a report
> > of recent regressions.
> >
> > The following bug entry is on the current list of known regressions
> > from 2.6.31.  Please verify if it still should be listed and let me know
> > (either way).
> >
> >
> > Bug-Entry       : http://bugzilla.kernel.org/show_bug.cgi?id=14379
> > Subject         : ACPI Warning for _SB_.BAT0._BIF: Converted Buffer to expected String
> > Submitter       : Justin Mattock <justinmattock-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
> > Date            : 2009-10-08 21:46 (4 days old)
> > References      : http://marc.info/?l=linux-kernel&m=125504031328941&w=4
> >
> >
> >
> 
> pulled rc4 today and the warning is still present.
> here is the results of git bisect:

Patch at http://bugzilla.kernel.org/show_bug.cgi?id=14379#c1 fixes the
warning.

Actually, the _BIF warning was fixed at commit
e899b6485c332aa2d7510739507ab5e5d7b28e59.
See http://bugzilla.kernel.org/show_bug.cgi?id=11822#c14 

I made a mistake at the cleanup of predefined method table so the
warning shows up again.

Lin Ming

> 
> d9adc2e031bd22d5d9607a53a8d3b30e0b675f39 is the first bad commit
> commit d9adc2e031bd22d5d9607a53a8d3b30e0b675f39
> Author: Lin Ming <ming.m.lin-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> Date:   Mon Jul 27 11:31:10 2009 +0800
> 
>     ACPICA: reformat predefined method table, no functional change
> 
>     Reformatted the methods that return package objects.
> 
>     Signed-off-by: Lin Ming <ming.m.lin-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>     Signed-off-by: Len Brown <len.brown-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> 
> :040000 040000 17c30a6e75ce2fafc677b32bdcf3fbecae8458ab
> f43e38a2a4034bcc73ef14943e61187f11b08731 M      drivers
> 
> I will go and attach the git log to the bug report.
> 

Re: [Bug #14379] ACPI Warning for _SB_.BAT0._BIF: Converted Buffer to expected String

[Justin P. Mattock]

Lin Ming wrote:
> On Mon, 2009-10-12 at 11:14 +0800, Justin Mattock wrote:
>    
>> On Sun, Oct 11, 2009 at 3:22 PM, Rafael J. Wysocki<rjw@sisk.pl>  wrote:
>>      
>>> This message has been generated automatically as a part of a report
>>> of recent regressions.
>>>
>>> The following bug entry is on the current list of known regressions
>>> from 2.6.31.  Please verify if it still should be listed and let me know
>>> (either way).
>>>
>>>
>>> Bug-Entry       : http://bugzilla.kernel.org/show_bug.cgi?id=14379
>>> Subject         : ACPI Warning for _SB_.BAT0._BIF: Converted Buffer to expected String
>>> Submitter       : Justin Mattock<justinmattock@gmail.com>
>>> Date            : 2009-10-08 21:46 (4 days old)
>>> References      : http://marc.info/?l=linux-kernel&m=125504031328941&w=4
>>>
>>>
>>>
>>>        
>> pulled rc4 today and the warning is still present.
>> here is the results of git bisect:
>>      
>
> Patch at http://bugzilla.kernel.org/show_bug.cgi?id=14379#c1 fixes the
> warning.
>
> Actually, the _BIF warning was fixed at commit
> e899b6485c332aa2d7510739507ab5e5d7b28e59.
> See http://bugzilla.kernel.org/show_bug.cgi?id=11822#c14
>
> I made a mistake at the cleanup of predefined method table so the
> warning shows up again.
>
> Lin Ming
>
>    
Cool, I can give that a try.
then I guess just leave this open(bugreport) until this patch
http://bugzilla.kernel.org/show_bug.cgi?id=14379#c1
gets pushed into the main tree then close the bugreport.

Justin P. Mattock

Re: [Bug #14370] ext4 corruptions

[Alexey Fisher]

Am Montag, den 12.10.2009, 00:22 +0200 schrieb Rafael J. Wysocki:
> This message has been generated automatically as a part of a report
> of recent regressions.
> 
> The following bug entry is on the current list of known regressions
> from 2.6.31.  Please verify if it still should be listed and let me know
> (either way).
> 
> 
> Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14370
> Subject		: ext4 corruptions
> Submitter	: Alexey Fisher <bug-track@fisher-privat.net>
> Date		: 2009-10-09 19:20 (3 days old)
> References	: http://marc.info/?l=linux-kernel&m=125511643504864&w=4
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe kernel-testers" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

This can be removed, it is duplicate of this bug:
http://bugzilla.kernel.org/show_bug.cgi?id=14354

Re: [Bug #14279] Suspend to RAM freeze totally since 2.6.32-rc1 - Acer Aspire 1511Lmi laptop

[Jan Beulich]

The issue is still present, albeit a patch (available in the bugzilla entry) had
been posted on Oct 6th. Jan

>>> "Rafael J. Wysocki" <rjw@sisk.pl> 12.10.09 00:22 >>>
This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.31.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14279 
Subject		: Suspend to RAM freeze totally since 2.6.32-rc1 - Acer Aspire 1511Lmi laptop
Submitter	: Christian Casteyde <casteyde.christian@free.fr>
Date		: 2009-09-30 18:14 (12 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=5f68563996e812f9ca35b3939ad2a42e5d254d66 

Re: [Bug #14378] Problems with net/core/skbuff.c

[David Miller]

From: "Rafael J. Wysocki" <rjw@sisk.pl>
Date: Mon, 12 Oct 2009 00:22:04 +0200 (CEST)

> Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14378
> Subject		: Problems with net/core/skbuff.c
> Submitter	: Massimo Cetra <mcetra@navynet.it>
> Date		: 2009-10-08 14:51 (4 days old)
> References	: http://marc.info/?l=linux-kernel&m=125501488220358&w=4

I don't know what to do about this one.

The user indicates that they have the vserver patches applied,
so maybe there is some interaction with that stuff.

Re: [Bug #14278] New message "NOHZ: local_softirq_pending 08" at each ping request

[Michael Buesch]

On Monday 12 October 2009 00:22:00 Rafael J. Wysocki wrote:
> This message has been generated automatically as a part of a report
> of recent regressions.
> 
> The following bug entry is on the current list of known regressions
> from 2.6.31.  Please verify if it still should be listed and let me know
> (either way).
> 
> 
> Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14278
> Subject		: New message "NOHZ: local_softirq_pending 08" at each ping request
> Submitter	: Christian Casteyde <casteyde.christian-GANU6spQydw@public.gmane.org>
> Date		: 2009-09-30 18:12 (12 days old)
> Handled-By	: Michael Buesch <mb-fseUSCV1ubazQB+pC5nmwQ@public.gmane.org>

I think John or Johannes is handling it, as my patch was rejected.

> Patch		: http://bugzilla.kernel.org/attachment.cgi?id=23220

-- 
Greetings, Michael.

Re: [Bug #14374] MCEs caused by commit db8be50c4307dac2b37305fc59c8dc0f978d09ea

[Mikael Pettersson]

Rafael J. Wysocki wrote:
> The following bug entry is on the current list of known regressions
> from 2.6.31.  Please verify if it still should be listed and let me know
> (either way).
> 
> 
> Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=3D14374
> Subject		: MCEs caused by commit db8be50c4307dac2b37305fc59c8dc0f978d09ea
> Submitter	: Nick Piggin <npiggin@suse.de>
> Date		: 2009-10-02 7:34 (10 days old)
> First-Bad-Commit: http://git.kernel.org/?p=3Dlinux/kernel/git/torvalds/linu=
> x-2.6.git;a=3Dcommit;h=3Ddb8be50c4307dac2b37305fc59c8dc0f978d09ea
> References	: http://marc.info/?l=3Dlinux-kernel&m=3D125446885705223&w=3D4

Fixed for 2.6.32-rc5 by d93a8f829fe1d2f3002f2c6ddb553d12db420412
'Revert "USB: Work around BIOS bugs by quiescing USB controllers earlier"'.

Re: [Bug #14378] Problems with net/core/skbuff.c

[Massimo "CtRiX" Cetra]

David Miller ha scritto:
> From: "Rafael J. Wysocki" <rjw@sisk.pl>
> Date: Mon, 12 Oct 2009 00:22:04 +0200 (CEST)
> 
>> Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14378
>> Subject		: Problems with net/core/skbuff.c
>> Submitter	: Massimo Cetra <mcetra@navynet.it>
>> Date		: 2009-10-08 14:51 (4 days old)
>> References	: http://marc.info/?l=linux-kernel&m=125501488220358&w=4
> 
> I don't know what to do about this one.
> 
> The user indicates that they have the vserver patches applied,
> so maybe there is some interaction with that stuff.

Actually i don't think that it's Vserver related.

I'm not a kernel hacker but the vserver patch doesn't seems to have much 
relation with networking (for who don't know vserver, it's a sort of 
advanced chroot).

I'm using OCFS2 and drbd. Both of those 2 things make heavy use of 
networking to share data between two hosts.

The backtrace seems to begin in the "world" of OFCS2.
I'm CC'ing ocfs mailing list where someome may have some clues.

Thanks,
Max

Re: [Bug #14372] Wireless not working after suspend-resume

[Fabio Comolli]

Still present with -rc4.

Error is:

[  690.547660] wlan0: direct probe to AP 00:11:95:09:2c:e0 (try 1)
[  690.550064] wlan0: direct probe responded
[  690.550074] wlan0: authenticate with AP 00:11:95:09:2c:e0 (try 1)
[  690.551629] wlan0: authenticated
[  690.551663] wlan0: associate with AP 00:11:95:09:2c:e0 (try 1)
[  690.553289] wlan0: RX AssocResp from 00:11:95:09:2c:e0 (capab=0x451
status=12 aid=1)
[  690.553298] wlan0: AP denied association (code=12)

rfkill-ing the device off and then on fixes the problem.

Fabio




On Mon, Oct 12, 2009 at 12:22 AM, Rafael J. Wysocki <rjw-KKrjLPT3xs0@public.gmane.org> wrote:
> This message has been generated automatically as a part of a report
> of recent regressions.
>
> The following bug entry is on the current list of known regressions
> from 2.6.31.  Please verify if it still should be listed and let me know
> (either way).
>
>
> Bug-Entry       : http://bugzilla.kernel.org/show_bug.cgi?id=14372
> Subject         : Wireless not working after suspend-resume
> Submitter       : Fabio Comolli <fabio.comolli@gmail.com>
> Date            : 2009-10-03 15:36 (9 days old)
> References      : http://lkml.org/lkml/2009/10/3/91
>
>
>

Re: [Bug #14278] New message "NOHZ: local_softirq_pending 08" at each ping request

[Rafael J. Wysocki]

On Monday 12 October 2009, Michael Buesch wrote:
> On Monday 12 October 2009 00:22:00 Rafael J. Wysocki wrote:
> > This message has been generated automatically as a part of a report
> > of recent regressions.
> > 
> > The following bug entry is on the current list of known regressions
> > from 2.6.31.  Please verify if it still should be listed and let me know
> > (either way).
> > 
> > 
> > Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14278
> > Subject		: New message "NOHZ: local_softirq_pending 08" at each ping request
> > Submitter	: Christian Casteyde <casteyde.christian-GANU6spQydw@public.gmane.org>
> > Date		: 2009-09-30 18:12 (12 days old)
> > Handled-By	: Michael Buesch <mb-fseUSCV1ubazQB+pC5nmwQ@public.gmane.org>
> 
> I think John or Johannes is handling it, as my patch was rejected.
> 
> > Patch		: http://bugzilla.kernel.org/attachment.cgi?id=23220

Johannes, what's the status here?

Rafael

Re: [Bug #14279] Suspend to RAM freeze totally since 2.6.32-rc1 - Acer Aspire 1511Lmi laptop

[Rafael J. Wysocki]

On Monday 12 October 2009, Jan Beulich wrote:
> The issue is still present, albeit a patch (available in the bugzilla entry) had
> been posted on Oct 6th. Jan

Thanks for the update.  I saw the patch go into the tip tree a few moments ago.

Best,
Rafael

Re: [Bug #14370] ext4 corruptions

[Rafael J. Wysocki]

On Monday 12 October 2009, Alexey Fisher wrote:
> Am Montag, den 12.10.2009, 00:22 +0200 schrieb Rafael J. Wysocki:
> > This message has been generated automatically as a part of a report
> > of recent regressions.
> > 
> > The following bug entry is on the current list of known regressions
> > from 2.6.31.  Please verify if it still should be listed and let me know
> > (either way).
> > 
> > 
> > Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14370
> > Subject		: ext4 corruptions
> > Submitter	: Alexey Fisher <bug-track-M18mAb7Tlt0yCq4wW13eYl6hYfS7NtTn@public.gmane.org>
> > Date		: 2009-10-09 19:20 (3 days old)
> > References	: http://marc.info/?l=linux-kernel&m=125511643504864&w=4
> > 
> > 
> > --
> > To unsubscribe from this list: send the line "unsubscribe kernel-testers" in
> > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> This can be removed, it is duplicate of this bug:
> http://bugzilla.kernel.org/show_bug.cgi?id=14354

Thanks, I closed #14370 as a duplicate of #14354.

Best,
Rafael

Re: [Bug #14372] Wireless not working after suspend-resume

[Rafael J. Wysocki]

On Monday 12 October 2009, Fabio Comolli wrote:
> Still present with -rc4.
> 
> Error is:
> 
> [  690.547660] wlan0: direct probe to AP 00:11:95:09:2c:e0 (try 1)
> [  690.550064] wlan0: direct probe responded
> [  690.550074] wlan0: authenticate with AP 00:11:95:09:2c:e0 (try 1)
> [  690.551629] wlan0: authenticated
> [  690.551663] wlan0: associate with AP 00:11:95:09:2c:e0 (try 1)
> [  690.553289] wlan0: RX AssocResp from 00:11:95:09:2c:e0 (capab=0x451
> status=12 aid=1)
> [  690.553298] wlan0: AP denied association (code=12)
> 
> rfkill-ing the device off and then on fixes the problem.

Thanks for the update.

Rafael

Re: [Bug #14374] MCEs caused by commit db8be50c4307dac2b37305fc59c8dc0f978d09ea

[Rafael J. Wysocki]

On Monday 12 October 2009, Mikael Pettersson wrote:
> Rafael J. Wysocki wrote:
> > The following bug entry is on the current list of known regressions
> > from 2.6.31.  Please verify if it still should be listed and let me know
> > (either way).
> > 
> > 
> > Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=3D14374
> > Subject		: MCEs caused by commit db8be50c4307dac2b37305fc59c8dc0f978d09ea
> > Submitter	: Nick Piggin <npiggin@suse.de>
> > Date		: 2009-10-02 7:34 (10 days old)
> > First-Bad-Commit: http://git.kernel.org/?p=3Dlinux/kernel/git/torvalds/linu=
> > x-2.6.git;a=3Dcommit;h=3Ddb8be50c4307dac2b37305fc59c8dc0f978d09ea
> > References	: http://marc.info/?l=3Dlinux-kernel&m=3D125446885705223&w=3D4
> 
> Fixed for 2.6.32-rc5 by d93a8f829fe1d2f3002f2c6ddb553d12db420412
> 'Revert "USB: Work around BIOS bugs by quiescing USB controllers earlier"'.

Thanks, closed.

Rafael

Re: [Bug #14379] ACPI Warning for _SB_.BAT0._BIF: Converted Buffer to expected String

[Rafael J. Wysocki]

On Monday 12 October 2009, Justin P. Mattock wrote:
> Lin Ming wrote:
> > On Mon, 2009-10-12 at 11:14 +0800, Justin Mattock wrote:
> >    
> >> On Sun, Oct 11, 2009 at 3:22 PM, Rafael J. Wysocki<rjw@sisk.pl>  wrote:
> >>      
> >>> This message has been generated automatically as a part of a report
> >>> of recent regressions.
> >>>
> >>> The following bug entry is on the current list of known regressions
> >>> from 2.6.31.  Please verify if it still should be listed and let me know
> >>> (either way).
> >>>
> >>>
> >>> Bug-Entry       : http://bugzilla.kernel.org/show_bug.cgi?id=14379
> >>> Subject         : ACPI Warning for _SB_.BAT0._BIF: Converted Buffer to expected String
> >>> Submitter       : Justin Mattock<justinmattock@gmail.com>
> >>> Date            : 2009-10-08 21:46 (4 days old)
> >>> References      : http://marc.info/?l=linux-kernel&m=125504031328941&w=4
> >>>
> >>>
> >>>
> >>>        
> >> pulled rc4 today and the warning is still present.
> >> here is the results of git bisect:
> >>      
> >
> > Patch at http://bugzilla.kernel.org/show_bug.cgi?id=14379#c1 fixes the
> > warning.
> >
> > Actually, the _BIF warning was fixed at commit
> > e899b6485c332aa2d7510739507ab5e5d7b28e59.
> > See http://bugzilla.kernel.org/show_bug.cgi?id=11822#c14
> >
> > I made a mistake at the cleanup of predefined method table so the
> > warning shows up again.
> >
> > Lin Ming
> >
> >    
> Cool, I can give that a try.
> then I guess just leave this open(bugreport) until this patch
> http://bugzilla.kernel.org/show_bug.cgi?id=14379#c1
> gets pushed into the main tree then close the bugreport.

I updated the bug entry with the patch information.

Thanks,
Rafael

Re: [Bug #14379] ACPI Warning for _SB_.BAT0._BIF: Converted Buffer to expected String

[Justin P. Mattock]

Rafael J. Wysocki wrote:
> On Monday 12 October 2009, Justin P. Mattock wrote:
>    
>> Lin Ming wrote:
>>      
>>> On Mon, 2009-10-12 at 11:14 +0800, Justin Mattock wrote:
>>>
>>>        
>>>> On Sun, Oct 11, 2009 at 3:22 PM, Rafael J. Wysocki<rjw-KKrjLPT3xs0@public.gmane.org>   wrote:
>>>>
>>>>          
>>>>> This message has been generated automatically as a part of a report
>>>>> of recent regressions.
>>>>>
>>>>> The following bug entry is on the current list of known regressions
>>>>> from 2.6.31.  Please verify if it still should be listed and let me know
>>>>> (either way).
>>>>>
>>>>>
>>>>> Bug-Entry       : http://bugzilla.kernel.org/show_bug.cgi?id=14379
>>>>> Subject         : ACPI Warning for _SB_.BAT0._BIF: Converted Buffer to expected String
>>>>> Submitter       : Justin Mattock<justinmattock-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
>>>>> Date            : 2009-10-08 21:46 (4 days old)
>>>>> References      : http://marc.info/?l=linux-kernel&m=125504031328941&w=4
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>            
>>>> pulled rc4 today and the warning is still present.
>>>> here is the results of git bisect:
>>>>
>>>>          
>>> Patch at http://bugzilla.kernel.org/show_bug.cgi?id=14379#c1 fixes the
>>> warning.
>>>
>>> Actually, the _BIF warning was fixed at commit
>>> e899b6485c332aa2d7510739507ab5e5d7b28e59.
>>> See http://bugzilla.kernel.org/show_bug.cgi?id=11822#c14
>>>
>>> I made a mistake at the cleanup of predefined method table so the
>>> warning shows up again.
>>>
>>> Lin Ming
>>>
>>>
>>>        
>> Cool, I can give that a try.
>> then I guess just leave this open(bugreport) until this patch
>> http://bugzilla.kernel.org/show_bug.cgi?id=14379#c1
>> gets pushed into the main tree then close the bugreport.
>>      
>
> I updated the bug entry with the patch information.
>
> Thanks,
> Rafael
>
>    
o.k. after somehow blowing my system up, I'm up and
running. Anyways I applied that patch and yes it works.

Justin P. Mattock

Re: [Bug #14372] Wireless not working after suspend-resume

[Maciej Rutecki]

CC added

2009/10/12 Fabio Comolli <fabio.comolli@gmail.com>:

[...]
> [  690.553298] wlan0: AP denied association (code=12)
[...]

I have the same code in different situation:
http://bugzilla.intellinuxwireless.org/show_bug.cgi?id=2089

Regards
-- 
Maciej Rutecki
http://www.maciek.unixy.pl

Re: [Bug #14378] Problems with net/core/skbuff.c

[Massimo Cetra]

[-- Attachment #1: Type: text/plain, Size: 959 bytes --]

David Miller ha scritto:
> From: "Rafael J. Wysocki" <rjw@sisk.pl>
> Date: Mon, 12 Oct 2009 00:22:04 +0200 (CEST)
>
>   
>> Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14378
>> Subject		: Problems with net/core/skbuff.c
>> Submitter	: Massimo Cetra <mcetra@navynet.it>
>> Date		: 2009-10-08 14:51 (4 days old)
>> References	: http://marc.info/?l=linux-kernel&m=125501488220358&w=4
>>     
>
> I don't know what to do about this one.
>
> The user indicates that they have the vserver patches applied,
> so maybe there is some interaction with that stuff.
>   
Actually i found another oops which is very similar to the previous one.
Here, vserver is not involved, and the problem starts at drbd which 
lives in kernel space (the other oops started at ocfs2).

Both ocfs2 and drbd make heavy use of network I/O so i guess the problem 
is something in the network layer.

Anything i can do to help to debugging and solving this issue ?

Thanks
Max




[-- Attachment #2: oops2.txt --]
[-- Type: text/plain, Size: 122916 bytes --]

[60257.728500] ------------[ cut here ]------------
[60257.728500] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8()
[60257.728500] Hardware name: 
[60257.728500] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys
[60257.728500] Pid: 1814, comm: drbd1_receiver Tainted: G        W  2.6.31.2-vserver-navynet #1
[60257.728500] Call Trace:
[60257.728500]  <IRQ>  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60257.728500]  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60257.728500]  [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3
[60257.728500]  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60257.728500]  [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d
[60257.728500]  [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net]
[60257.728500]  [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net]
[60257.728500]  [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205
[60257.728500]  [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole]
[60257.728500]  [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f
[60257.728500]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60257.728500]  [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba
[60257.728500]  [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b
[60257.728500]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60257.728500]  [<ffffffff81308309>] ? printk+0x4e/0x5d
[60257.728500]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60257.728500]  [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf
[60257.728500]  [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49
[60257.728500]  [<ffffffff810626b7>] ? ktime_get+0xc/0x41
[60257.728500]  [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146
[60257.728500]  [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93
[60257.728500]  [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20
[60257.728500]  <EOI>  [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3
[60257.728500]  [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31
[60257.728500]  [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec
[60257.728500]  [<ffffffff81308f73>] ? thread_return+0x47/0xd5
[60257.728500]  [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a
[60257.728500]  [<ffffffff81309213>] ? schedule_timeout+0x21/0x197
[60257.728500]  [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10
[60257.728500]  [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29
[60257.728500]  [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3
[60257.728500]  [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca
[60257.728500]  [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60257.728500]  [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac
[60257.728500]  [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd
[60257.728500]  [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5
[60257.728500]  [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45
[60257.728500]  [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d
[60257.728500]  [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60257.728500]  [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41
[60257.728500]  [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd]
[60257.728500]  [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276
[60257.728500]  [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd]
[60257.728500]  [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd]
[60257.728500]  [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd]
[60257.728500]  [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd]
[60257.728500]  [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd]
[60257.728500]  [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd]
[60257.728500]  [<ffffffff81011b9a>] ? child_rip+0xa/0x20
[60257.728500]  [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4
[60257.728500]  [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd]
[60257.728500]  [<ffffffff81011b90>] ? child_rip+0x0/0x20
[60257.728500] ---[ end trace 694817acca794f2c ]---
[60257.728500] ------------[ cut here ]------------
[60257.728500] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8()
[60257.728500] Hardware name: 
[60257.728500] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys
[60257.728500] Pid: 1814, comm: drbd1_receiver Tainted: G        W  2.6.31.2-vserver-navynet #1
[60257.728500] Call Trace:
[60257.728500]  <IRQ>  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60257.728500]  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60257.728500]  [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3
[60257.728500]  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60257.728500]  [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d
[60257.728500]  [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net]
[60257.728500]  [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net]
[60257.728500]  [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205
[60257.728500]  [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole]
[60257.728500]  [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f
[60257.728500]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60257.728500]  [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba
[60257.728500]  [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b
[60257.728500]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60257.728500]  [<ffffffff81308309>] ? printk+0x4e/0x5d
[60257.728500]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60257.728500]  [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf
[60257.728500]  [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49
[60257.728500]  [<ffffffff810626b7>] ? ktime_get+0xc/0x41
[60257.728500]  [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146
[60257.728500]  [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93
[60257.728500]  [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20
[60257.728500]  <EOI>  [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3
[60257.728500]  [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31
[60257.728500]  [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec
[60257.728500]  [<ffffffff81308f73>] ? thread_return+0x47/0xd5
[60257.728500]  [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a
[60257.728500]  [<ffffffff81309213>] ? schedule_timeout+0x21/0x197
[60257.728500]  [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10
[60257.728500]  [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29
[60257.728500]  [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3
[60257.728500]  [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca
[60257.728500]  [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60257.728500]  [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac
[60257.728500]  [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd
[60257.728500]  [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5
[60257.728500]  [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45
[60257.728500]  [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d
[60257.728500]  [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60257.728500]  [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41
[60257.728500]  [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd]
[60257.728500]  [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276
[60257.728500]  [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd]
[60257.728500]  [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd]
[60257.728500]  [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd]
[60257.728500]  [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd]
[60257.728500]  [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd]
[60257.728500]  [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd]
[60257.728500]  [<ffffffff81011b9a>] ? child_rip+0xa/0x20
[60257.728500]  [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4
[60257.728500]  [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd]
[60257.728500]  [<ffffffff81011b90>] ? child_rip+0x0/0x20
[60257.728500] ---[ end trace 694817acca794f2d ]---
[60257.728500] ------------[ cut here ]------------
[60257.728500] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8()
[60257.728500] Hardware name: 
[60257.728500] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys
[60257.728500] Pid: 1814, comm: drbd1_receiver Tainted: G        W  2.6.31.2-vserver-navynet #1
[60257.728500] Call Trace:
[60257.728500]  <IRQ>  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60257.728500]  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60257.728500]  [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3
[60257.728500]  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60257.728500]  [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d
[60257.728500]  [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net]
[60257.728500]  [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net]
[60257.728500]  [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205
[60257.728500]  [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole]
[60257.728500]  [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f
[60257.728500]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60257.728500]  [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba
[60257.728500]  [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b
[60257.728500]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60257.728500]  [<ffffffff81308309>] ? printk+0x4e/0x5d
[60257.728500]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60257.728500]  [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf
[60257.728500]  [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49
[60257.728500]  [<ffffffff810626b7>] ? ktime_get+0xc/0x41
[60257.728500]  [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146
[60257.728500]  [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93
[60257.728500]  [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20
[60257.728500]  <EOI>  [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3
[60257.728500]  [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31
[60257.728500]  [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec
[60257.728500]  [<ffffffff81308f73>] ? thread_return+0x47/0xd5
[60257.728500]  [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a
[60257.728500]  [<ffffffff81309213>] ? schedule_timeout+0x21/0x197
[60257.728500]  [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10
[60257.728500]  [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29
[60257.728500]  [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3
[60257.728500]  [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca
[60257.728500]  [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60257.728500]  [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac
[60257.728500]  [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd
[60257.728500]  [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5
[60257.728500]  [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45
[60257.728500]  [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d
[60257.728500]  [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60257.728500]  [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41
[60257.728500]  [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd]
[60257.728500]  [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276
[60257.728500]  [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd]
[60257.728500]  [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd]
[60257.728500]  [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd]
[60257.728500]  [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd]
[60257.728500]  [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd]
[60257.728500]  [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd]
[60257.728500]  [<ffffffff81011b9a>] ? child_rip+0xa/0x20
[60257.728500]  [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4
[60257.728500]  [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd]
[60257.728500]  [<ffffffff81011b90>] ? child_rip+0x0/0x20
[60257.728500] ---[ end trace 694817acca794f2e ]---
[60257.728500] ------------[ cut here ]------------
[60257.728500] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8()
[60257.728500] Hardware name: 
[60257.728500] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys
[60257.728500] Pid: 1814, comm: drbd1_receiver Tainted: G        W  2.6.31.2-vserver-navynet #1
[60257.728500] Call Trace:
[60257.728500]  <IRQ>  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60257.728500]  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60257.728500]  [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3
[60257.728500]  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60257.728500]  [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d
[60257.728500]  [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net]
[60257.728500]  [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net]
[60257.728500]  [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205
[60257.728500]  [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole]
[60257.728500]  [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f
[60257.728500]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60257.728500]  [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba
[60257.728500]  [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b
[60257.728500]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60257.728500]  [<ffffffff81308309>] ? printk+0x4e/0x5d
[60257.728500]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60257.728500]  [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf
[60257.728500]  [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49
[60257.728500]  [<ffffffff810626b7>] ? ktime_get+0xc/0x41
[60257.728500]  [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146
[60257.728500]  [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93
[60257.728500]  [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20
[60257.728500]  <EOI>  [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3
[60257.728500]  [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31
[60257.728500]  [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec
[60257.728500]  [<ffffffff81308f73>] ? thread_return+0x47/0xd5
[60257.728500]  [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a
[60257.728500]  [<ffffffff81309213>] ? schedule_timeout+0x21/0x197
[60257.728500]  [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10
[60257.728500]  [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29
[60257.728500]  [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3
[60257.728500]  [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca
[60257.728500]  [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60257.728500]  [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac
[60257.728500]  [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd
[60257.728500]  [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5
[60257.728500]  [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45
[60257.728500]  [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d
[60257.728500]  [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60257.728500]  [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41
[60257.728500]  [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd]
[60257.728500]  [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276
[60257.728500]  [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd]
[60257.728500]  [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd]
[60257.728500]  [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd]
[60257.728500]  [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd]
[60257.728500]  [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd]
[60257.728500]  [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd]
[60257.728500]  [<ffffffff81011b9a>] ? child_rip+0xa/0x20
[60257.728500]  [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4
[60257.728500]  [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd]
[60257.728500]  [<ffffffff81011b90>] ? child_rip+0x0/0x20
[60257.728500] ---[ end trace 694817acca794f2f ]---
[60257.728500] ------------[ cut here ]------------
[60257.728500] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8()
[60257.728500] Hardware name: 
[60257.728500] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys
[60257.728500] Pid: 1814, comm: drbd1_receiver Tainted: G        W  2.6.31.2-vserver-navynet #1
[60257.728500] Call Trace:
[60257.728500]  <IRQ>  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60257.728500]  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60257.728500]  [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3
[60257.728500]  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60257.728500]  [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d
[60257.728500]  [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net]
[60257.728500]  [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net]
[60257.728500]  [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205
[60257.728500]  [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole]
[60257.728500]  [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f
[60257.728500]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60257.728500]  [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba
[60257.728500]  [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b
[60257.728500]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60257.728500]  [<ffffffff81308309>] ? printk+0x4e/0x5d
[60257.728500]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60257.728500]  [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf
[60257.728500]  [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49
[60257.728500]  [<ffffffff810626b7>] ? ktime_get+0xc/0x41
[60257.728500]  [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146
[60257.728500]  [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93
[60257.728500]  [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20
[60257.728500]  <EOI>  [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3
[60257.728500]  [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31
[60257.728500]  [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec
[60257.728500]  [<ffffffff81308f73>] ? thread_return+0x47/0xd5
[60257.728500]  [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a
[60257.728500]  [<ffffffff81309213>] ? schedule_timeout+0x21/0x197
[60257.728500]  [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10
[60257.728500]  [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29
[60257.728500]  [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3
[60257.728500]  [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca
[60257.728500]  [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60257.728500]  [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac
[60257.728500]  [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd
[60257.728500]  [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5
[60257.728500]  [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45
[60257.728500]  [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d
[60257.728500]  [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60257.728500]  [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41
[60257.728500]  [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd]
[60257.728500]  [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276
[60257.728500]  [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd]
[60257.728500]  [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd]
[60257.728500]  [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd]
[60257.728500]  [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd]
[60257.728500]  [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd]
[60257.728500]  [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd]
[60257.728500]  [<ffffffff81011b9a>] ? child_rip+0xa/0x20
[60257.728500]  [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4
[60257.728500]  [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd]
[60257.728500]  [<ffffffff81011b90>] ? child_rip+0x0/0x20
[60257.728500] ---[ end trace 694817acca794f30 ]---
[60257.728500] ------------[ cut here ]------------
[60257.728500] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8()
[60257.728500] Hardware name: 
[60257.728500] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys
[60257.728500] Pid: 1814, comm: drbd1_receiver Tainted: G        W  2.6.31.2-vserver-navynet #1
[60257.728500] Call Trace:
[60257.728500]  <IRQ>  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60257.728500]  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60257.728500]  [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3
[60257.728500]  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60257.728500]  [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d
[60257.728500]  [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net]
[60257.728500]  [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net]
[60257.728500]  [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205
[60257.728500]  [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole]
[60257.728500]  [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f
[60257.728500]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60257.728500]  [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba
[60257.728500]  [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b
[60257.728500]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60257.728500]  [<ffffffff81308309>] ? printk+0x4e/0x5d
[60257.728500]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60257.728500]  [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf
[60257.728500]  [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49
[60257.728500]  [<ffffffff810626b7>] ? ktime_get+0xc/0x41
[60257.728500]  [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146
[60257.728500]  [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93
[60257.728500]  [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20
[60257.728500]  <EOI>  [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3
[60257.728500]  [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31
[60257.728500]  [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec
[60257.728500]  [<ffffffff81308f73>] ? thread_return+0x47/0xd5
[60257.728500]  [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a
[60257.728500]  [<ffffffff81309213>] ? schedule_timeout+0x21/0x197
[60257.728500]  [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10
[60257.728500]  [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29
[60257.728500]  [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3
[60257.728500]  [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca
[60257.728500]  [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60257.728500]  [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac
[60257.728500]  [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd
[60257.728500]  [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5
[60257.728500]  [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45
[60257.728500]  [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d
[60257.728500]  [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60257.728500]  [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41
[60257.728500]  [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd]
[60257.728500]  [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276
[60257.728500]  [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd]
[60257.728500]  [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd]
[60257.728500]  [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd]
[60257.728500]  [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd]
[60257.728500]  [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd]
[60257.728500]  [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd]
[60257.728500]  [<ffffffff81011b9a>] ? child_rip+0xa/0x20
[60257.728500]  [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4
[60257.728500]  [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd]
[60257.728500]  [<ffffffff81011b90>] ? child_rip+0x0/0x20
[60257.728500] ---[ end trace 694817acca794f31 ]---
[60257.728500] ------------[ cut here ]------------
[60257.728500] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8()
[60257.728500] Hardware name: 
[60257.728500] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys
[60257.728500] Pid: 1814, comm: drbd1_receiver Tainted: G        W  2.6.31.2-vserver-navynet #1
[60257.728500] Call Trace:
[60257.728500]  <IRQ>  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60257.728500]  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60257.728500]  [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3
[60257.728500]  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60257.728500]  [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d
[60257.728500]  [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net]
[60257.728500]  [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net]
[60257.728500]  [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205
[60257.728500]  [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole]
[60257.728500]  [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f
[60257.728500]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60257.728500]  [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba
[60257.728500]  [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b
[60257.728500]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60257.728500]  [<ffffffff81308309>] ? printk+0x4e/0x5d
[60257.728500]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60257.728500]  [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf
[60257.728500]  [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49
[60257.728500]  [<ffffffff810626b7>] ? ktime_get+0xc/0x41
[60257.728500]  [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146
[60257.728500]  [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93
[60257.728500]  [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20
[60257.728500]  <EOI>  [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3
[60257.728500]  [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31
[60257.728500]  [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec
[60257.728500]  [<ffffffff81308f73>] ? thread_return+0x47/0xd5
[60257.728500]  [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a
[60257.728500]  [<ffffffff81309213>] ? schedule_timeout+0x21/0x197
[60257.728500]  [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10
[60257.728500]  [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29
[60257.728500]  [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3
[60257.728500]  [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca
[60257.728500]  [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60257.728500]  [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac
[60257.728500]  [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd
[60257.728500]  [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5
[60257.728500]  [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45
[60257.728500]  [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d
[60257.728500]  [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60257.728500]  [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41
[60257.728500]  [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd]
[60257.728500]  [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276
[60257.728500]  [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd]
[60257.728500]  [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd]
[60257.728500]  [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd]
[60257.728500]  [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd]
[60257.728500]  [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd]
[60257.728500]  [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd]
[60257.728500]  [<ffffffff81011b9a>] ? child_rip+0xa/0x20
[60257.728500]  [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4
[60257.728500]  [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd]
[60257.728500]  [<ffffffff81011b90>] ? child_rip+0x0/0x20
[60257.728500] ---[ end trace 694817acca794f32 ]---
[60257.728500] ------------[ cut here ]------------
[60257.728500] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8()
[60257.728500] Hardware name: 
[60257.728500] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys
[60257.728500] Pid: 1814, comm: drbd1_receiver Tainted: G        W  2.6.31.2-vserver-navynet #1
[60257.728500] Call Trace:
[60257.728500]  <IRQ>  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60257.728500]  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60257.728500]  [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3
[60257.728500]  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60257.728500]  [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d
[60257.728500]  [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net]
[60257.728500]  [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net]
[60257.728500]  [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205
[60257.728500]  [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole]
[60257.728500]  [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f
[60257.728500]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60257.728500]  [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba
[60257.728500]  [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b
[60257.728500]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60257.728500]  [<ffffffff81308309>] ? printk+0x4e/0x5d
[60257.728500]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60257.728500]  [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf
[60257.728500]  [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49
[60257.728500]  [<ffffffff810626b7>] ? ktime_get+0xc/0x41
[60257.728500]  [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146
[60257.728500]  [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93
[60257.728500]  [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20
[60257.728500]  <EOI>  [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3
[60257.728500]  [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31
[60257.728500]  [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec
[60257.728500]  [<ffffffff81308f73>] ? thread_return+0x47/0xd5
[60257.728500]  [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a
[60257.728500]  [<ffffffff81309213>] ? schedule_timeout+0x21/0x197
[60257.728500]  [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10
[60257.728500]  [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29
[60257.728500]  [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3
[60257.728500]  [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca
[60257.728500]  [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60257.728500]  [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac
[60257.728500]  [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd
[60257.728500]  [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5
[60257.728500]  [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45
[60257.728500]  [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d
[60257.728500]  [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60257.728500]  [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41
[60257.728500]  [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd]
[60257.728500]  [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276
[60257.728500]  [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd]
[60257.728500]  [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd]
[60257.728500]  [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd]
[60257.728500]  [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd]
[60257.728500]  [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd]
[60257.728500]  [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd]
[60257.728500]  [<ffffffff81011b9a>] ? child_rip+0xa/0x20
[60257.728500]  [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4
[60257.728500]  [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd]
[60257.728500]  [<ffffffff81011b90>] ? child_rip+0x0/0x20
[60257.728500] ---[ end trace 694817acca794f33 ]---
[60257.728500] ------------[ cut here ]------------
[60257.728500] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8()
[60257.728500] Hardware name: 
[60257.728500] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys
[60257.728500] Pid: 1814, comm: drbd1_receiver Tainted: G        W  2.6.31.2-vserver-navynet #1
[60257.728500] Call Trace:
[60257.728500]  <IRQ>  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60257.728500]  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60257.728500]  [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3
[60257.728500]  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60257.728500]  [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d
[60257.728500]  [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net]
[60257.728500]  [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net]
[60257.728500]  [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205
[60257.728500]  [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole]
[60257.728500]  [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f
[60257.728500]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60257.728500]  [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba
[60257.728500]  [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b
[60257.728500]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60257.728500]  [<ffffffff81308309>] ? printk+0x4e/0x5d
[60257.728500]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60257.728500]  [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf
[60257.728500]  [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49
[60257.728500]  [<ffffffff810626b7>] ? ktime_get+0xc/0x41
[60257.728500]  [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146
[60257.728500]  [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93
[60257.728500]  [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20
[60257.728500]  <EOI>  [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3
[60257.728500]  [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31
[60257.728500]  [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec
[60257.728500]  [<ffffffff81308f73>] ? thread_return+0x47/0xd5
[60257.728500]  [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a
[60257.728500]  [<ffffffff81309213>] ? schedule_timeout+0x21/0x197
[60257.728500]  [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10
[60257.728500]  [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29
[60257.728500]  [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3
[60257.728500]  [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca
[60257.728500]  [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60257.728500]  [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac
[60257.728500]  [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd
[60257.728500]  [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5
[60257.728500]  [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45
[60257.728500]  [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d
[60257.728500]  [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60257.728500]  [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41
[60257.728500]  [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd]
[60257.728500]  [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276
[60257.728500]  [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd]
[60257.728500]  [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd]
[60257.728500]  [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd]
[60257.728500]  [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd]
[60257.728500]  [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd]
[60257.728500]  [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd]
[60257.728500]  [<ffffffff81011b9a>] ? child_rip+0xa/0x20
[60257.728500]  [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4
[60257.728500]  [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd]
[60257.728500]  [<ffffffff81011b90>] ? child_rip+0x0/0x20
[60257.728500] ---[ end trace 694817acca794f34 ]---
[60257.728500] ------------[ cut here ]------------
[60257.728500] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8()
[60257.728500] Hardware name: 
[60257.728500] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys
[60257.728500] Pid: 1814, comm: drbd1_receiver Tainted: G        W  2.6.31.2-vserver-navynet #1
[60257.728500] Call Trace:
[60257.728500]  <IRQ>  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60257.728500]  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60257.728500]  [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3
[60257.728500]  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60257.728500]  [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d
[60257.728500]  [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net]
[60257.728500]  [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net]
[60257.728500]  [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205
[60257.728500]  [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole]
[60257.728500]  [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f
[60257.728500]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60257.728500]  [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba
[60257.728500]  [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b
[60257.728500]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60257.728500]  [<ffffffff81308309>] ? printk+0x4e/0x5d
[60257.728500]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60257.728500]  [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf
[60257.728500]  [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49
[60257.728500]  [<ffffffff810626b7>] ? ktime_get+0xc/0x41
[60257.728500]  [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146
[60257.728500]  [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93
[60257.728500]  [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20
[60257.728500]  <EOI>  [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3
[60257.728500]  [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31
[60257.728500]  [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec
[60257.728500]  [<ffffffff81308f73>] ? thread_return+0x47/0xd5
[60257.728500]  [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a
[60257.728500]  [<ffffffff81309213>] ? schedule_timeout+0x21/0x197
[60257.728500]  [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10
[60257.728500]  [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29
[60257.728500]  [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3
[60257.728500]  [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca
[60257.728500]  [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60257.728500]  [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac
[60257.728500]  [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd
[60257.728500]  [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5
[60257.728500]  [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45
[60257.728500]  [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d
[60257.728500]  [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60257.728500]  [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41
[60257.728500]  [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd]
[60257.728500]  [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276
[60257.728500]  [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd]
[60257.728500]  [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd]
[60257.728500]  [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd]
[60257.728500]  [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd]
[60257.728500]  [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd]
[60257.728500]  [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd]
[60257.728500]  [<ffffffff81011b9a>] ? child_rip+0xa/0x20
[60257.728500]  [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4
[60257.728500]  [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd]
[60257.728500]  [<ffffffff81011b90>] ? child_rip+0x0/0x20
[60257.728500] ---[ end trace 694817acca794f35 ]---
[60258.036152] ------------[ cut here ]------------
[60258.036152] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8()
[60258.036152] Hardware name: 
[60258.036152] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys
[60258.036152] Pid: 1814, comm: drbd1_receiver Tainted: G        W  2.6.31.2-vserver-navynet #1
[60258.036152] Call Trace:
[60258.036152]  <IRQ>  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60258.036152]  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60258.036152]  [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3
[60258.036152]  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60258.036152]  [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d
[60258.036152]  [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net]
[60258.036152]  [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net]
[60258.036152]  [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205
[60258.036152]  [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole]
[60258.036152]  [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f
[60258.036152]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60258.036152]  [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba
[60258.036152]  [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b
[60258.036152]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60258.036152]  [<ffffffff81308309>] ? printk+0x4e/0x5d
[60258.036152]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60258.036152]  [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf
[60258.036152]  [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49
[60258.036152]  [<ffffffff810626b7>] ? ktime_get+0xc/0x41
[60258.036152]  [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146
[60258.036152]  [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93
[60258.036152]  [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20
[60258.036152]  <EOI>  [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3
[60258.036152]  [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31
[60258.036152]  [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec
[60258.036152]  [<ffffffff81308f73>] ? thread_return+0x47/0xd5
[60258.036152]  [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a
[60258.036152]  [<ffffffff81309213>] ? schedule_timeout+0x21/0x197
[60258.036152]  [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10
[60258.036152]  [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29
[60258.036152]  [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3
[60258.036152]  [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca
[60258.036152]  [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60258.036152]  [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac
[60258.036152]  [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd
[60258.036152]  [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5
[60258.036152]  [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45
[60258.036152]  [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d
[60258.036152]  [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60258.036152]  [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41
[60258.036152]  [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd]
[60258.036152]  [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276
[60258.036152]  [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd]
[60258.036152]  [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd]
[60258.036152]  [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd]
[60258.036152]  [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd]
[60258.036152]  [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd]
[60258.036152]  [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd]
[60258.036152]  [<ffffffff81011b9a>] ? child_rip+0xa/0x20
[60258.036152]  [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4
[60258.036152]  [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd]
[60258.036152]  [<ffffffff81011b90>] ? child_rip+0x0/0x20
[60258.036152] ---[ end trace 694817acca794f36 ]---
[60258.036152] ------------[ cut here ]------------
[60258.036152] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8()
[60258.036152] Hardware name: 
[60258.036152] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys
[60258.036152] Pid: 1814, comm: drbd1_receiver Tainted: G        W  2.6.31.2-vserver-navynet #1
[60258.036152] Call Trace:
[60258.036152]  <IRQ>  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60258.036152]  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60258.036152]  [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3
[60258.036152]  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60258.036152]  [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d
[60258.036152]  [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net]
[60258.036152]  [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net]
[60258.036152]  [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205
[60258.036152]  [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole]
[60258.036152]  [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f
[60258.036152]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60258.036152]  [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba
[60258.036152]  [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b
[60258.036152]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60258.036152]  [<ffffffff81308309>] ? printk+0x4e/0x5d
[60258.036152]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60258.036152]  [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf
[60258.036152]  [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49
[60258.036152]  [<ffffffff810626b7>] ? ktime_get+0xc/0x41
[60258.036152]  [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146
[60258.036152]  [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93
[60258.036152]  [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20
[60258.036152]  <EOI>  [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3
[60258.036152]  [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31
[60258.036152]  [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec
[60258.036152]  [<ffffffff81308f73>] ? thread_return+0x47/0xd5
[60258.036152]  [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a
[60258.036152]  [<ffffffff81309213>] ? schedule_timeout+0x21/0x197
[60258.036152]  [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10
[60258.036152]  [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29
[60258.036152]  [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3
[60258.036152]  [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca
[60258.036152]  [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60258.036152]  [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac
[60258.036152]  [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd
[60258.036152]  [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5
[60258.036152]  [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45
[60258.036152]  [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d
[60258.036152]  [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60258.036152]  [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41
[60258.036152]  [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd]
[60258.036152]  [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276
[60258.036152]  [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd]
[60258.036152]  [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd]
[60258.036152]  [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd]
[60258.036152]  [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd]
[60258.036152]  [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd]
[60258.036152]  [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd]
[60258.036152]  [<ffffffff81011b9a>] ? child_rip+0xa/0x20
[60258.036152]  [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4
[60258.036152]  [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd]
[60258.036152]  [<ffffffff81011b90>] ? child_rip+0x0/0x20
[60258.036152] ---[ end trace 694817acca794f37 ]---
[60258.036152] ------------[ cut here ]------------
[60258.036152] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8()
[60258.036152] Hardware name: 
[60258.036152] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys
[60258.036152] Pid: 1814, comm: drbd1_receiver Tainted: G        W  2.6.31.2-vserver-navynet #1
[60258.036152] Call Trace:
[60258.036152]  <IRQ>  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60258.036152]  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60258.036152]  [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3
[60258.036152]  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60258.036152]  [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d
[60258.036152]  [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net]
[60258.036152]  [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net]
[60258.036152]  [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205
[60258.036152]  [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole]
[60258.036152]  [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f
[60258.036152]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60258.036152]  [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba
[60258.036152]  [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b
[60258.036152]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60258.036152]  [<ffffffff81308309>] ? printk+0x4e/0x5d
[60258.036152]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60258.036152]  [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf
[60258.036152]  [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49
[60258.036152]  [<ffffffff810626b7>] ? ktime_get+0xc/0x41
[60258.036152]  [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146
[60258.036152]  [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93
[60258.036152]  [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20
[60258.036152]  <EOI>  [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3
[60258.036152]  [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31
[60258.036152]  [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec
[60258.036152]  [<ffffffff81308f73>] ? thread_return+0x47/0xd5
[60258.036152]  [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a
[60258.036152]  [<ffffffff81309213>] ? schedule_timeout+0x21/0x197
[60258.036152]  [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10
[60258.036152]  [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29
[60258.036152]  [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3
[60258.036152]  [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca
[60258.036152]  [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60258.036152]  [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac
[60258.036152]  [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd
[60258.036152]  [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5
[60258.036152]  [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45
[60258.036152]  [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d
[60258.036152]  [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60258.036152]  [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41
[60258.036152]  [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd]
[60258.036152]  [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276
[60258.036152]  [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd]
[60258.036152]  [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd]
[60258.036152]  [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd]
[60258.036152]  [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd]
[60258.036152]  [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd]
[60258.036152]  [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd]
[60258.036152]  [<ffffffff81011b9a>] ? child_rip+0xa/0x20
[60258.036152]  [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4
[60258.036152]  [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd]
[60258.036152]  [<ffffffff81011b90>] ? child_rip+0x0/0x20
[60258.036152] ---[ end trace 694817acca794f38 ]---
[60258.036152] ------------[ cut here ]------------
[60258.036152] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8()
[60258.036152] Hardware name: 
[60258.036152] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys
[60258.036152] Pid: 1814, comm: drbd1_receiver Tainted: G        W  2.6.31.2-vserver-navynet #1
[60258.036152] Call Trace:
[60258.036152]  <IRQ>  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60258.036152]  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60258.036152]  [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3
[60258.036152]  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60258.036152]  [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d
[60258.036152]  [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net]
[60258.036152]  [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net]
[60258.036152]  [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205
[60258.036152]  [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole]
[60258.036152]  [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f
[60258.036152]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60258.036152]  [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba
[60258.036152]  [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b
[60258.036152]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60258.036152]  [<ffffffff81308309>] ? printk+0x4e/0x5d
[60258.036152]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60258.036152]  [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf
[60258.036152]  [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49
[60258.036152]  [<ffffffff810626b7>] ? ktime_get+0xc/0x41
[60258.036152]  [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146
[60258.036152]  [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93
[60258.036152]  [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20
[60258.036152]  <EOI>  [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3
[60258.036152]  [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31
[60258.036152]  [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec
[60258.036152]  [<ffffffff81308f73>] ? thread_return+0x47/0xd5
[60258.036152]  [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a
[60258.036152]  [<ffffffff81309213>] ? schedule_timeout+0x21/0x197
[60258.036152]  [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10
[60258.036152]  [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29
[60258.036152]  [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3
[60258.036152]  [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca
[60258.036152]  [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60258.036152]  [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac
[60258.036152]  [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd
[60258.036152]  [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5
[60258.036152]  [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45
[60258.036152]  [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d
[60258.036152]  [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60258.036152]  [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41
[60258.036152]  [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd]
[60258.036152]  [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276
[60258.036152]  [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd]
[60258.036152]  [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd]
[60258.036152]  [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd]
[60258.036152]  [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd]
[60258.036152]  [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd]
[60258.036152]  [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd]
[60258.036152]  [<ffffffff81011b9a>] ? child_rip+0xa/0x20
[60258.036152]  [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4
[60258.036152]  [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd]
[60258.036152]  [<ffffffff81011b90>] ? child_rip+0x0/0x20
[60258.036152] ---[ end trace 694817acca794f39 ]---
[60258.036152] ------------[ cut here ]------------
[60258.036152] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8()
[60258.036152] Hardware name: 
[60258.036152] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys
[60258.036152] Pid: 1814, comm: drbd1_receiver Tainted: G        W  2.6.31.2-vserver-navynet #1
[60258.036152] Call Trace:
[60258.036152]  <IRQ>  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60258.036152]  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60258.036152]  [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3
[60258.036152]  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60258.036152]  [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d
[60258.036152]  [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net]
[60258.036152]  [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net]
[60258.036152]  [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205
[60258.036152]  [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole]
[60258.036152]  [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f
[60258.036152]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60258.036152]  [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba
[60258.036152]  [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b
[60258.036152]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60258.036152]  [<ffffffff81308309>] ? printk+0x4e/0x5d
[60258.036152]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60258.036152]  [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf
[60258.036152]  [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49
[60258.036152]  [<ffffffff810626b7>] ? ktime_get+0xc/0x41
[60258.036152]  [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146
[60258.036152]  [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93
[60258.036152]  [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20
[60258.036152]  <EOI>  [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3
[60258.036152]  [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31
[60258.036152]  [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec
[60258.036152]  [<ffffffff81308f73>] ? thread_return+0x47/0xd5
[60258.036152]  [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a
[60258.036152]  [<ffffffff81309213>] ? schedule_timeout+0x21/0x197
[60258.036152]  [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10
[60258.036152]  [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29
[60258.036152]  [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3
[60258.036152]  [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca
[60258.036152]  [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60258.036152]  [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac
[60258.036152]  [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd
[60258.036152]  [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5
[60258.036152]  [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45
[60258.036152]  [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d
[60258.036152]  [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60258.036152]  [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41
[60258.036152]  [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd]
[60258.036152]  [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276
[60258.036152]  [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd]
[60258.036152]  [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd]
[60258.036152]  [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd]
[60258.036152]  [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd]
[60258.036152]  [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd]
[60258.036152]  [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd]
[60258.036152]  [<ffffffff81011b9a>] ? child_rip+0xa/0x20
[60258.036152]  [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4
[60258.036152]  [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd]
[60258.036152]  [<ffffffff81011b90>] ? child_rip+0x0/0x20
[60258.036152] ---[ end trace 694817acca794f3a ]---
[60258.036152] ------------[ cut here ]------------
[60258.036152] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8()
[60258.036152] Hardware name: 
[60258.036152] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys
[60258.036152] Pid: 1814, comm: drbd1_receiver Tainted: G        W  2.6.31.2-vserver-navynet #1
[60258.036152] Call Trace:
[60258.036152]  <IRQ>  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60258.036152]  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60258.036152]  [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3
[60258.036152]  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60258.036152]  [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d
[60258.036152]  [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net]
[60258.036152]  [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net]
[60258.036152]  [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205
[60258.036152]  [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole]
[60258.036152]  [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f
[60258.036152]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60258.036152]  [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba
[60258.036152]  [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b
[60258.036152]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60258.036152]  [<ffffffff81308309>] ? printk+0x4e/0x5d
[60258.036152]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60258.036152]  [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf
[60258.036152]  [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49
[60258.036152]  [<ffffffff810626b7>] ? ktime_get+0xc/0x41
[60258.036152]  [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146
[60258.036152]  [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93
[60258.036152]  [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20
[60258.036152]  <EOI>  [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3
[60258.036152]  [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31
[60258.036152]  [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec
[60258.036152]  [<ffffffff81308f73>] ? thread_return+0x47/0xd5
[60258.036152]  [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a
[60258.036152]  [<ffffffff81309213>] ? schedule_timeout+0x21/0x197
[60258.036152]  [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10
[60258.036152]  [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29
[60258.036152]  [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3
[60258.036152]  [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca
[60258.036152]  [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60258.036152]  [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac
[60258.036152]  [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd
[60258.036152]  [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5
[60258.036152]  [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45
[60258.036152]  [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d
[60258.036152]  [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60258.036152]  [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41
[60258.036152]  [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd]
[60258.036152]  [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276
[60258.036152]  [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd]
[60258.036152]  [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd]
[60258.036152]  [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd]
[60258.036152]  [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd]
[60258.036152]  [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd]
[60258.036152]  [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd]
[60258.036152]  [<ffffffff81011b9a>] ? child_rip+0xa/0x20
[60258.036152]  [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4
[60258.036152]  [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd]
[60258.036152]  [<ffffffff81011b90>] ? child_rip+0x0/0x20
[60258.036152] ---[ end trace 694817acca794f3b ]---
[60258.036152] ------------[ cut here ]------------
[60258.036152] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8()
[60258.036152] Hardware name: 
[60258.036152] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys
[60258.036152] Pid: 1814, comm: drbd1_receiver Tainted: G        W  2.6.31.2-vserver-navynet #1
[60258.036152] Call Trace:
[60258.036152]  <IRQ>  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60258.036152]  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60258.036152]  [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3
[60258.036152]  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60258.036152]  [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d
[60258.036152]  [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net]
[60258.036152]  [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net]
[60258.036152]  [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205
[60258.036152]  [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole]
[60258.036152]  [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f
[60258.036152]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60258.036152]  [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba
[60258.036152]  [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b
[60258.036152]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60258.036152]  [<ffffffff81308309>] ? printk+0x4e/0x5d
[60258.036152]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60258.036152]  [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf
[60258.036152]  [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49
[60258.036152]  [<ffffffff810626b7>] ? ktime_get+0xc/0x41
[60258.036152]  [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146
[60258.036152]  [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93
[60258.036152]  [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20
[60258.036152]  <EOI>  [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3
[60258.036152]  [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31
[60258.036152]  [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec
[60258.036152]  [<ffffffff81308f73>] ? thread_return+0x47/0xd5
[60258.036152]  [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a
[60258.036152]  [<ffffffff81309213>] ? schedule_timeout+0x21/0x197
[60258.036152]  [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10
[60258.036152]  [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29
[60258.036152]  [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3
[60258.036152]  [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca
[60258.036152]  [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60258.036152]  [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac
[60258.036152]  [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd
[60258.036152]  [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5
[60258.036152]  [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45
[60258.036152]  [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d
[60258.036152]  [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60258.036152]  [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41
[60258.036152]  [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd]
[60258.036152]  [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276
[60258.036152]  [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd]
[60258.036152]  [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd]
[60258.036152]  [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd]
[60258.036152]  [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd]
[60258.036152]  [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd]
[60258.036152]  [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd]
[60258.036152]  [<ffffffff81011b9a>] ? child_rip+0xa/0x20
[60258.036152]  [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4
[60258.036152]  [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd]
[60258.036152]  [<ffffffff81011b90>] ? child_rip+0x0/0x20
[60258.036152] ---[ end trace 694817acca794f3c ]---
[60259.116162] ------------[ cut here ]------------
[60259.116162] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8()
[60259.116162] Hardware name: 
[60259.116162] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys
[60259.116162] Pid: 1814, comm: drbd1_receiver Tainted: G        W  2.6.31.2-vserver-navynet #1
[60259.116162] Call Trace:
[60259.116162]  <IRQ>  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162]  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162]  [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3
[60259.116162]  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162]  [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d
[60259.116162]  [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net]
[60259.116162]  [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net]
[60259.116162]  [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205
[60259.116162]  [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole]
[60259.116162]  [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f
[60259.116162]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60259.116162]  [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba
[60259.116162]  [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b
[60259.116162]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60259.116162]  [<ffffffff81308309>] ? printk+0x4e/0x5d
[60259.116162]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60259.116162]  [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf
[60259.116162]  [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49
[60259.116162]  [<ffffffff810626b7>] ? ktime_get+0xc/0x41
[60259.116162]  [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146
[60259.116162]  [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93
[60259.116162]  [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20
[60259.116162]  <EOI>  [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3
[60259.116162]  [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31
[60259.116162]  [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec
[60259.116162]  [<ffffffff81308f73>] ? thread_return+0x47/0xd5
[60259.116162]  [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a
[60259.116162]  [<ffffffff81309213>] ? schedule_timeout+0x21/0x197
[60259.116162]  [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10
[60259.116162]  [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29
[60259.116162]  [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3
[60259.116162]  [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca
[60259.116162]  [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60259.116162]  [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac
[60259.116162]  [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd
[60259.116162]  [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5
[60259.116162]  [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45
[60259.116162]  [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d
[60259.116162]  [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60259.116162]  [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41
[60259.116162]  [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd]
[60259.116162]  [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276
[60259.116162]  [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd]
[60259.116162]  [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd]
[60259.116162]  [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd]
[60259.116162]  [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd]
[60259.116162]  [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd]
[60259.116162]  [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd]
[60259.116162]  [<ffffffff81011b9a>] ? child_rip+0xa/0x20
[60259.116162]  [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4
[60259.116162]  [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd]
[60259.116162]  [<ffffffff81011b90>] ? child_rip+0x0/0x20
[60259.116162] ---[ end trace 694817acca794f3d ]---
[60259.116162] ------------[ cut here ]------------
[60259.116162] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8()
[60259.116162] Hardware name: 
[60259.116162] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys
[60259.116162] Pid: 1814, comm: drbd1_receiver Tainted: G        W  2.6.31.2-vserver-navynet #1
[60259.116162] Call Trace:
[60259.116162]  <IRQ>  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162]  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162]  [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3
[60259.116162]  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162]  [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d
[60259.116162]  [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net]
[60259.116162]  [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net]
[60259.116162]  [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205
[60259.116162]  [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole]
[60259.116162]  [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f
[60259.116162]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60259.116162]  [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba
[60259.116162]  [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b
[60259.116162]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60259.116162]  [<ffffffff81308309>] ? printk+0x4e/0x5d
[60259.116162]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60259.116162]  [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf
[60259.116162]  [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49
[60259.116162]  [<ffffffff810626b7>] ? ktime_get+0xc/0x41
[60259.116162]  [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146
[60259.116162]  [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93
[60259.116162]  [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20
[60259.116162]  <EOI>  [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3
[60259.116162]  [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31
[60259.116162]  [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec
[60259.116162]  [<ffffffff81308f73>] ? thread_return+0x47/0xd5
[60259.116162]  [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a
[60259.116162]  [<ffffffff81309213>] ? schedule_timeout+0x21/0x197
[60259.116162]  [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10
[60259.116162]  [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29
[60259.116162]  [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3
[60259.116162]  [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca
[60259.116162]  [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60259.116162]  [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac
[60259.116162]  [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd
[60259.116162]  [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5
[60259.116162]  [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45
[60259.116162]  [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d
[60259.116162]  [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60259.116162]  [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41
[60259.116162]  [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd]
[60259.116162]  [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276
[60259.116162]  [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd]
[60259.116162]  [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd]
[60259.116162]  [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd]
[60259.116162]  [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd]
[60259.116162]  [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd]
[60259.116162]  [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd]
[60259.116162]  [<ffffffff81011b9a>] ? child_rip+0xa/0x20
[60259.116162]  [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4
[60259.116162]  [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd]
[60259.116162]  [<ffffffff81011b90>] ? child_rip+0x0/0x20
[60259.116162] ---[ end trace 694817acca794f3e ]---
[60259.116162] ------------[ cut here ]------------
[60259.116162] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8()
[60259.116162] Hardware name: 
[60259.116162] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys
[60259.116162] Pid: 1814, comm: drbd1_receiver Tainted: G        W  2.6.31.2-vserver-navynet #1
[60259.116162] Call Trace:
[60259.116162]  <IRQ>  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162]  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162]  [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3
[60259.116162]  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162]  [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d
[60259.116162]  [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net]
[60259.116162]  [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net]
[60259.116162]  [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205
[60259.116162]  [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole]
[60259.116162]  [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f
[60259.116162]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60259.116162]  [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba
[60259.116162]  [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b
[60259.116162]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60259.116162]  [<ffffffff81308309>] ? printk+0x4e/0x5d
[60259.116162]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60259.116162]  [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf
[60259.116162]  [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49
[60259.116162]  [<ffffffff810626b7>] ? ktime_get+0xc/0x41
[60259.116162]  [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146
[60259.116162]  [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93
[60259.116162]  [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20
[60259.116162]  <EOI>  [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3
[60259.116162]  [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31
[60259.116162]  [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec
[60259.116162]  [<ffffffff81308f73>] ? thread_return+0x47/0xd5
[60259.116162]  [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a
[60259.116162]  [<ffffffff81309213>] ? schedule_timeout+0x21/0x197
[60259.116162]  [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10
[60259.116162]  [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29
[60259.116162]  [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3
[60259.116162]  [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca
[60259.116162]  [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60259.116162]  [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac
[60259.116162]  [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd
[60259.116162]  [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5
[60259.116162]  [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45
[60259.116162]  [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d
[60259.116162]  [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60259.116162]  [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41
[60259.116162]  [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd]
[60259.116162]  [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276
[60259.116162]  [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd]
[60259.116162]  [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd]
[60259.116162]  [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd]
[60259.116162]  [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd]
[60259.116162]  [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd]
[60259.116162]  [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd]
[60259.116162]  [<ffffffff81011b9a>] ? child_rip+0xa/0x20
[60259.116162]  [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4
[60259.116162]  [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd]
[60259.116162]  [<ffffffff81011b90>] ? child_rip+0x0/0x20
[60259.116162] ---[ end trace 694817acca794f3f ]---
[60259.116162] ------------[ cut here ]------------
[60259.116162] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8()
[60259.116162] Hardware name: 
[60259.116162] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys
[60259.116162] Pid: 1814, comm: drbd1_receiver Tainted: G        W  2.6.31.2-vserver-navynet #1
[60259.116162] Call Trace:
[60259.116162]  <IRQ>  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162]  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162]  [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3
[60259.116162]  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162]  [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d
[60259.116162]  [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net]
[60259.116162]  [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net]
[60259.116162]  [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205
[60259.116162]  [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole]
[60259.116162]  [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f
[60259.116162]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60259.116162]  [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba
[60259.116162]  [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b
[60259.116162]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60259.116162]  [<ffffffff81308309>] ? printk+0x4e/0x5d
[60259.116162]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60259.116162]  [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf
[60259.116162]  [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49
[60259.116162]  [<ffffffff810626b7>] ? ktime_get+0xc/0x41
[60259.116162]  [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146
[60259.116162]  [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93
[60259.116162]  [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20
[60259.116162]  <EOI>  [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3
[60259.116162]  [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31
[60259.116162]  [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec
[60259.116162]  [<ffffffff81308f73>] ? thread_return+0x47/0xd5
[60259.116162]  [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a
[60259.116162]  [<ffffffff81309213>] ? schedule_timeout+0x21/0x197
[60259.116162]  [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10
[60259.116162]  [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29
[60259.116162]  [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3
[60259.116162]  [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca
[60259.116162]  [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60259.116162]  [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac
[60259.116162]  [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd
[60259.116162]  [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5
[60259.116162]  [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45
[60259.116162]  [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d
[60259.116162]  [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60259.116162]  [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41
[60259.116162]  [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd]
[60259.116162]  [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276
[60259.116162]  [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd]
[60259.116162]  [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd]
[60259.116162]  [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd]
[60259.116162]  [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd]
[60259.116162]  [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd]
[60259.116162]  [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd]
[60259.116162]  [<ffffffff81011b9a>] ? child_rip+0xa/0x20
[60259.116162]  [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4
[60259.116162]  [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd]
[60259.116162]  [<ffffffff81011b90>] ? child_rip+0x0/0x20
[60259.116162] ---[ end trace 694817acca794f40 ]---
[60259.116162] ------------[ cut here ]------------
[60259.116162] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8()
[60259.116162] Hardware name: 
[60259.116162] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys
[60259.116162] Pid: 1814, comm: drbd1_receiver Tainted: G        W  2.6.31.2-vserver-navynet #1
[60259.116162] Call Trace:
[60259.116162]  <IRQ>  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162]  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162]  [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3
[60259.116162]  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162]  [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d
[60259.116162]  [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net]
[60259.116162]  [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net]
[60259.116162]  [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205
[60259.116162]  [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole]
[60259.116162]  [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f
[60259.116162]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60259.116162]  [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba
[60259.116162]  [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b
[60259.116162]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60259.116162]  [<ffffffff81308309>] ? printk+0x4e/0x5d
[60259.116162]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60259.116162]  [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf
[60259.116162]  [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49
[60259.116162]  [<ffffffff810626b7>] ? ktime_get+0xc/0x41
[60259.116162]  [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146
[60259.116162]  [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93
[60259.116162]  [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20
[60259.116162]  <EOI>  [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3
[60259.116162]  [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31
[60259.116162]  [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec
[60259.116162]  [<ffffffff81308f73>] ? thread_return+0x47/0xd5
[60259.116162]  [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a
[60259.116162]  [<ffffffff81309213>] ? schedule_timeout+0x21/0x197
[60259.116162]  [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10
[60259.116162]  [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29
[60259.116162]  [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3
[60259.116162]  [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca
[60259.116162]  [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60259.116162]  [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac
[60259.116162]  [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd
[60259.116162]  [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5
[60259.116162]  [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45
[60259.116162]  [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d
[60259.116162]  [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60259.116162]  [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41
[60259.116162]  [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd]
[60259.116162]  [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276
[60259.116162]  [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd]
[60259.116162]  [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd]
[60259.116162]  [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd]
[60259.116162]  [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd]
[60259.116162]  [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd]
[60259.116162]  [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd]
[60259.116162]  [<ffffffff81011b9a>] ? child_rip+0xa/0x20
[60259.116162]  [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4
[60259.116162]  [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd]
[60259.116162]  [<ffffffff81011b90>] ? child_rip+0x0/0x20
[60259.116162] ---[ end trace 694817acca794f41 ]---
[60259.116162] ------------[ cut here ]------------
[60259.116162] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8()
[60259.116162] Hardware name: 
[60259.116162] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys
[60259.116162] Pid: 1814, comm: drbd1_receiver Tainted: G        W  2.6.31.2-vserver-navynet #1
[60259.116162] Call Trace:
[60259.116162]  <IRQ>  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162]  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162]  [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3
[60259.116162]  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162]  [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d
[60259.116162]  [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net]
[60259.116162]  [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net]
[60259.116162]  [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205
[60259.116162]  [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole]
[60259.116162]  [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f
[60259.116162]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60259.116162]  [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba
[60259.116162]  [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b
[60259.116162]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60259.116162]  [<ffffffff81308309>] ? printk+0x4e/0x5d
[60259.116162]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60259.116162]  [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf
[60259.116162]  [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49
[60259.116162]  [<ffffffff810626b7>] ? ktime_get+0xc/0x41
[60259.116162]  [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146
[60259.116162]  [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93
[60259.116162]  [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20
[60259.116162]  <EOI>  [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3
[60259.116162]  [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31
[60259.116162]  [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec
[60259.116162]  [<ffffffff81308f73>] ? thread_return+0x47/0xd5
[60259.116162]  [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a
[60259.116162]  [<ffffffff81309213>] ? schedule_timeout+0x21/0x197
[60259.116162]  [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10
[60259.116162]  [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29
[60259.116162]  [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3
[60259.116162]  [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca
[60259.116162]  [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60259.116162]  [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac
[60259.116162]  [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd
[60259.116162]  [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5
[60259.116162]  [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45
[60259.116162]  [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d
[60259.116162]  [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60259.116162]  [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41
[60259.116162]  [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd]
[60259.116162]  [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276
[60259.116162]  [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd]
[60259.116162]  [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd]
[60259.116162]  [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd]
[60259.116162]  [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd]
[60259.116162]  [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd]
[60259.116162]  [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd]
[60259.116162]  [<ffffffff81011b9a>] ? child_rip+0xa/0x20
[60259.116162]  [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4
[60259.116162]  [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd]
[60259.116162]  [<ffffffff81011b90>] ? child_rip+0x0/0x20
[60259.116162] ---[ end trace 694817acca794f42 ]---
[60259.116162] ------------[ cut here ]------------
[60259.116162] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8()
[60259.116162] Hardware name: 
[60259.116162] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys
[60259.116162] Pid: 1814, comm: drbd1_receiver Tainted: G        W  2.6.31.2-vserver-navynet #1
[60259.116162] Call Trace:
[60259.116162]  <IRQ>  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162]  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162]  [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3
[60259.116162]  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162]  [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d
[60259.116162]  [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net]
[60259.116162]  [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net]
[60259.116162]  [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205
[60259.116162]  [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole]
[60259.116162]  [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f
[60259.116162]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60259.116162]  [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba
[60259.116162]  [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b
[60259.116162]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60259.116162]  [<ffffffff81308309>] ? printk+0x4e/0x5d
[60259.116162]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60259.116162]  [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf
[60259.116162]  [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49
[60259.116162]  [<ffffffff810626b7>] ? ktime_get+0xc/0x41
[60259.116162]  [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146
[60259.116162]  [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93
[60259.116162]  [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20
[60259.116162]  <EOI>  [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3
[60259.116162]  [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31
[60259.116162]  [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec
[60259.116162]  [<ffffffff81308f73>] ? thread_return+0x47/0xd5
[60259.116162]  [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a
[60259.116162]  [<ffffffff81309213>] ? schedule_timeout+0x21/0x197
[60259.116162]  [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10
[60259.116162]  [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29
[60259.116162]  [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3
[60259.116162]  [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca
[60259.116162]  [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60259.116162]  [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac
[60259.116162]  [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd
[60259.116162]  [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5
[60259.116162]  [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45
[60259.116162]  [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d
[60259.116162]  [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60259.116162]  [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41
[60259.116162]  [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd]
[60259.116162]  [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276
[60259.116162]  [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd]
[60259.116162]  [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd]
[60259.116162]  [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd]
[60259.116162]  [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd]
[60259.116162]  [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd]
[60259.116162]  [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd]
[60259.116162]  [<ffffffff81011b9a>] ? child_rip+0xa/0x20
[60259.116162]  [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4
[60259.116162]  [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd]
[60259.116162]  [<ffffffff81011b90>] ? child_rip+0x0/0x20
[60259.116162] ---[ end trace 694817acca794f43 ]---
[60259.116162] ------------[ cut here ]------------
[60259.116162] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8()
[60259.116162] Hardware name: 
[60259.116162] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys
[60259.116162] Pid: 1814, comm: drbd1_receiver Tainted: G        W  2.6.31.2-vserver-navynet #1
[60259.116162] Call Trace:
[60259.116162]  <IRQ>  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162]  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162]  [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3
[60259.116162]  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162]  [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d
[60259.116162]  [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net]
[60259.116162]  [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net]
[60259.116162]  [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205
[60259.116162]  [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole]
[60259.116162]  [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f
[60259.116162]  [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba
[60259.116162]  [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b
[60259.116162]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60259.116162]  [<ffffffff81308309>] ? printk+0x4e/0x5d
[60259.116162]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60259.116162]  [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf
[60259.116162]  [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49
[60259.116162]  [<ffffffff810626b7>] ? ktime_get+0xc/0x41
[60259.116162]  [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146
[60259.116162]  [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93
[60259.116162]  [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20
[60259.116162]  <EOI>  [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3
[60259.116162]  [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31
[60259.116162]  [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec
[60259.116162]  [<ffffffff81308f73>] ? thread_return+0x47/0xd5
[60259.116162]  [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a
[60259.116162]  [<ffffffff81309213>] ? schedule_timeout+0x21/0x197
[60259.116162]  [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10
[60259.116162]  [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29
[60259.116162]  [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3
[60259.116162]  [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca
[60259.116162]  [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60259.116162]  [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac
[60259.116162]  [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd
[60259.116162]  [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5
[60259.116162]  [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45
[60259.116162]  [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d
[60259.116162]  [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60259.116162]  [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41
[60259.116162]  [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd]
[60259.116162]  [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276
[60259.116162]  [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd]
[60259.116162]  [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd]
[60259.116162]  [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd]
[60259.116162]  [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd]
[60259.116162]  [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd]
[60259.116162]  [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd]
[60259.116162]  [<ffffffff81011b9a>] ? child_rip+0xa/0x20
[60259.116162]  [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4
[60259.116162]  [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd]
[60259.116162]  [<ffffffff81011b90>] ? child_rip+0x0/0x20
[60259.116162] ---[ end trace 694817acca794f44 ]---
[60259.116162] ------------[ cut here ]------------
[60259.116162] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8()
[60259.116162] Hardware name: 
[60259.116162] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys
[60259.116162] Pid: 1814, comm: drbd1_receiver Tainted: G        W  2.6.31.2-vserver-navynet #1
[60259.116162] Call Trace:
[60259.116162]  <IRQ>  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162]  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162]  [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3
[60259.116162]  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162]  [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d
[60259.116162]  [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net]
[60259.116162]  [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net]
[60259.116162]  [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205
[60259.116162]  [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole]
[60259.116162]  [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f
[60259.116162]  [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba
[60259.116162]  [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b
[60259.116162]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60259.116162]  [<ffffffff81308309>] ? printk+0x4e/0x5d
[60259.116162]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60259.116162]  [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf
[60259.116162]  [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49
[60259.116162]  [<ffffffff810626b7>] ? ktime_get+0xc/0x41
[60259.116162]  [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146
[60259.116162]  [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93
[60259.116162]  [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20
[60259.116162]  <EOI>  [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3
[60259.116162]  [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31
[60259.116162]  [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec
[60259.116162]  [<ffffffff81308f73>] ? thread_return+0x47/0xd5
[60259.116162]  [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a
[60259.116162]  [<ffffffff81309213>] ? schedule_timeout+0x21/0x197
[60259.116162]  [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10
[60259.116162]  [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29
[60259.116162]  [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3
[60259.116162]  [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca
[60259.116162]  [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60259.116162]  [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac
[60259.116162]  [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd
[60259.116162]  [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5
[60259.116162]  [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45
[60259.116162]  [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d
[60259.116162]  [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60259.116162]  [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41
[60259.116162]  [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd]
[60259.116162]  [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276
[60259.116162]  [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd]
[60259.116162]  [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd]
[60259.116162]  [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd]
[60259.116162]  [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd]
[60259.116162]  [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd]
[60259.116162]  [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd]
[60259.116162]  [<ffffffff81011b9a>] ? child_rip+0xa/0x20
[60259.116162]  [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4
[60259.116162]  [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd]
[60259.116162]  [<ffffffff81011b90>] ? child_rip+0x0/0x20
[60259.116162] ---[ end trace 694817acca794f45 ]---
[60259.116162] ------------[ cut here ]------------
[60259.116162] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8()
[60259.116162] Hardware name: 
[60259.116162] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys
[60259.116162] Pid: 1814, comm: drbd1_receiver Tainted: G        W  2.6.31.2-vserver-navynet #1
[60259.116162] Call Trace:
[60259.116162]  <IRQ>  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162]  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162]  [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3
[60259.116162]  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162]  [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d
[60259.116162]  [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net]
[60259.116162]  [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net]
[60259.116162]  [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205
[60259.116162]  [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole]
[60259.116162]  [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f
[60259.116162]  [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba
[60259.116162]  [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b
[60259.116162]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60259.116162]  [<ffffffff81308309>] ? printk+0x4e/0x5d
[60259.116162]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60259.116162]  [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf
[60259.116162]  [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49
[60259.116162]  [<ffffffff810626b7>] ? ktime_get+0xc/0x41
[60259.116162]  [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146
[60259.116162]  [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93
[60259.116162]  [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20
[60259.116162]  <EOI>  [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3
[60259.116162]  [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31
[60259.116162]  [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec
[60259.116162]  [<ffffffff81308f73>] ? thread_return+0x47/0xd5
[60259.116162]  [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a
[60259.116162]  [<ffffffff81309213>] ? schedule_timeout+0x21/0x197
[60259.116162]  [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10
[60259.116162]  [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29
[60259.116162]  [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3
[60259.116162]  [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca
[60259.116162]  [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60259.116162]  [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac
[60259.116162]  [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd
[60259.116162]  [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5
[60259.116162]  [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45
[60259.116162]  [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d
[60259.116162]  [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60259.116162]  [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41
[60259.116162]  [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd]
[60259.116162]  [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276
[60259.116162]  [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd]
[60259.116162]  [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd]
[60259.116162]  [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd]
[60259.116162]  [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd]
[60259.116162]  [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd]
[60259.116162]  [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd]
[60259.116162]  [<ffffffff81011b9a>] ? child_rip+0xa/0x20
[60259.116162]  [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4
[60259.116162]  [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd]
[60259.116162]  [<ffffffff81011b90>] ? child_rip+0x0/0x20
[60259.116162] ---[ end trace 694817acca794f46 ]---
[60259.116162] ------------[ cut here ]------------
[60259.116162] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8()
[60259.116162] Hardware name: 
[60259.116162] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys
[60259.116162] Pid: 1814, comm: drbd1_receiver Tainted: G        W  2.6.31.2-vserver-navynet #1
[60259.116162] Call Trace:
[60259.116162]  <IRQ>  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162]  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162]  [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3
[60259.116162]  [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
[60259.116162]  [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d
[60259.116162]  [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net]
[60259.116162]  [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net]
[60259.116162]  [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205
[60259.116162]  [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole]
[60259.116162]  [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f
[60259.116162]  [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba
[60259.116162]  [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b
[60259.116162]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60259.116162]  [<ffffffff81308309>] ? printk+0x4e/0x5d
[60259.116162]  [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
[60259.116162]  [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf
[60259.116162]  [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49
[60259.116162]  [<ffffffff810626b7>] ? ktime_get+0xc/0x41
[60259.116162]  [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146
[60259.116162]  [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93
[60259.116162]  [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20
[60259.116162]  <EOI>  [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3
[60259.116162]  [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31
[60259.116162]  [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec
[60259.116162]  [<ffffffff81308f73>] ? thread_return+0x47/0xd5
[60259.116162]  [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a
[60259.116162]  [<ffffffff81309213>] ? schedule_timeout+0x21/0x197
[60259.116162]  [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10
[60259.116162]  [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29
[60259.116162]  [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3
[60259.116162]  [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca
[60259.116162]  [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60259.116162]  [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac
[60259.116162]  [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd
[60259.116162]  [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5
[60259.116162]  [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45
[60259.116162]  [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d
[60259.116162]  [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
[60259.116162]  [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41
[60259.116162]  [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd]
[60259.116162]  [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276
[60259.116162]  [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd]
[60259.116162]  [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd]
[60259.116162]  [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd]
[60259.116162]  [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd]
[60259.116162]  [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd]
[60259.116162]  [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd]
[60259.116162]  [<ffffffff81011b9a>] ? child_rip+0xa/0x20
[60259.116162]  [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4
[60259.116162]  [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd]
[60259.116162]  [<ffffffff81011b90>] ? child_rip+0x0/0x20
[60259.116162] ---[ end trace 694817acca794f47 ]---

Re: [Bug #14378] Problems with net/core/skbuff.c

[Massimo Cetra]

Massimo Cetra ha scritto:
> David Miller ha scritto:
>> From: "Rafael J. Wysocki" <rjw@sisk.pl>
>> Date: Mon, 12 Oct 2009 00:22:04 +0200 (CEST)
>>
>>  
>>> Bug-Entry    : http://bugzilla.kernel.org/show_bug.cgi?id=14378
>>> Subject        : Problems with net/core/skbuff.c
>>> Submitter    : Massimo Cetra <mcetra@navynet.it>
>>> Date        : 2009-10-08 14:51 (4 days old)
>>> References    : http://marc.info/?l=linux-kernel&m=125501488220358&w=4
>>>     
>>
>> I don't know what to do about this one.
>>
>> The user indicates that they have the vserver patches applied,
>> so maybe there is some interaction with that stuff.
>>   
> Actually i found another oops which is very similar to the previous one.

And here it is another one, this time triggered by postfix, where mor 
drbd nor vserver are involved.
This is not the same server where the other oopses were grabbed.

Max

Re: [Bug #14378] Problems with net/core/skbuff.c

[Eric Dumazet]

Massimo Cetra a écrit :
> David Miller ha scritto:
>> From: "Rafael J. Wysocki" <rjw@sisk.pl>
>> Date: Mon, 12 Oct 2009 00:22:04 +0200 (CEST)
>>
>>  
>>> Bug-Entry    : http://bugzilla.kernel.org/show_bug.cgi?id=14378
>>> Subject        : Problems with net/core/skbuff.c
>>> Submitter    : Massimo Cetra <mcetra@navynet.it>
>>> Date        : 2009-10-08 14:51 (4 days old)
>>> References    : http://marc.info/?l=linux-kernel&m=125501488220358&w=4
>>>     
>>
>> I don't know what to do about this one.
>>
>> The user indicates that they have the vserver patches applied,
>> so maybe there is some interaction with that stuff.
>>   
> Actually i found another oops which is very similar to the previous one.
> Here, vserver is not involved, and the problem starts at drbd which
> lives in kernel space (the other oops started at ocfs2).
> 
> Both ocfs2 and drbd make heavy use of network I/O so i guess the problem
> is something in the network layer.
> 
> Anything i can do to help to debugging and solving this issue ?
> 
> Thanks
> Max
> 

Problem is kfree_skb() is called from irq context, wich is not allowed.

static void skb_release_head_state(struct sk_buff *skb)
{
...
if (skb->destructor) {
	WARN_ON(in_irq());
	skb->destructor();
}
...
}

virtio_net start_xmit() function calls free_old_xmit_skbs() and 
free_old_xmit_skbs() ultimately calls kfree_skb()

Quick fix would be to use dev_kfree_skb_any() instead,
because netpoll can definitly calls start_xmit()
with irq disabled.

Could you please following patch ?

diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index 8d00976..54bf091 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c
@@ -454,7 +454,7 @@ static unsigned int free_old_xmit_skbs(struct virtnet_info *vi)
 		vi->dev->stats.tx_bytes += skb->len;
 		vi->dev->stats.tx_packets++;
 		tot_sgs += skb_vnet_hdr(skb)->num_sg;
-		kfree_skb(skb);
+		dev_kfree_skb_any(skb);
 	}
 	return tot_sgs;
 }

Re: [Bug #14378] Problems with net/core/skbuff.c

[David Miller]

From: Massimo Cetra <ctrixk-BBpJ+9iBSNKonA0d6jMUrA@public.gmane.org>
Date: Tue, 13 Oct 2009 11:11:17 +0200

> Here, vserver is not involved, and the problem starts at drbd which
> lives in kernel space (the other oops started at ocfs2).
> 
> Both ocfs2 and drbd make heavy use of network I/O so i guess the
> problem is something in the network layer.
> 
> Anything i can do to help to debugging and solving this issue ?

Is all of your traffic going over virtio_net?

Re: [Bug #14378] Problems with net/core/skbuff.c

[David Miller]

From: Eric Dumazet <eric.dumazet@gmail.com>
Date: Tue, 13 Oct 2009 12:19:51 +0200

> Quick fix would be to use dev_kfree_skb_any() instead,
> because netpoll can definitly calls start_xmit()
> with irq disabled.

Indeed, because of netpoll(), that is something drivers
must be able to cope with.

> Could you please following patch ?

Indeed, let us know how Eric's patch works.

Thanks Eric!

Re: [Bug #14378] Problems with net/core/skbuff.c

[Massimo Cetra]

David Miller ha scritto:
> From: Massimo Cetra <ctrixk-BBpJ+9iBSNKonA0d6jMUrA@public.gmane.org>
> Date: Tue, 13 Oct 2009 11:11:17 +0200
>
>   
>> Here, vserver is not involved, and the problem starts at drbd which
>> lives in kernel space (the other oops started at ocfs2).
>>
>> Both ocfs2 and drbd make heavy use of network I/O so i guess the
>> problem is something in the network layer.
>>
>> Anything i can do to help to debugging and solving this issue ?
>>     
>
> Is all of your traffic going over virtio_net?
>   
Yes, indeed.
I'm compiling the patch and i'll let you know.

Thanks,
Max

Re: [Bug #14278] New message "NOHZ: local_softirq_pending 08" at each ping request

[John W. Linville]

On Mon, Oct 12, 2009 at 11:49:39PM +0200, Rafael J. Wysocki wrote:
> On Monday 12 October 2009, Michael Buesch wrote:
> > On Monday 12 October 2009 00:22:00 Rafael J. Wysocki wrote:
> > > This message has been generated automatically as a part of a report
> > > of recent regressions.
> > > 
> > > The following bug entry is on the current list of known regressions
> > > from 2.6.31.  Please verify if it still should be listed and let me know
> > > (either way).
> > > 
> > > 
> > > Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14278
> > > Subject		: New message "NOHZ: local_softirq_pending 08" at each ping request
> > > Submitter	: Christian Casteyde <casteyde.christian-GANU6spQydw@public.gmane.org>
> > > Date		: 2009-09-30 18:12 (12 days old)
> > > Handled-By	: Michael Buesch <mb-fseUSCV1ubazQB+pC5nmwQ@public.gmane.org>
> > 
> > I think John or Johannes is handling it, as my patch was rejected.
> > 
> > > Patch		: http://bugzilla.kernel.org/attachment.cgi?id=23220
> 
> Johannes, what's the status here?

Johannes posted a patch here:

	http://marc.info/?l=linux-wireless&m=125525657120789&w=2

It is on my merge train, but it hasn't gone to Linus (or Dave) just
yet -- probably today.

Hth!

John
-- 
John W. Linville		Someday the world will need a hero, and you
linville-2XuSBdqkA4R54TAoqtyWWQ@public.gmane.org			might be all we have.  Be ready.

Re: [Bug #14389] Build system issue

[Sam Ravnborg]

On Mon, Oct 12, 2009 at 12:22:09AM +0200, Rafael J. Wysocki wrote:
> This message has been generated automatically as a part of a report
> of recent regressions.
> 
> The following bug entry is on the current list of known regressions
> from 2.6.31.  Please verify if it still should be listed and let me know
> (either way).
> 
> 
> Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14389
> Subject		: Build system issue
> Submitter	: Peter Zijlstra <peterz-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>
> Date		: 2009-10-09 8:58 (3 days old)
> First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=575543347b5baed0ca927cb90ba8807396fe9cc9
> References	: http://marc.info/?l=linux-kernel&m=125507914909152&w=4

It is fixed by: 2331d1a6cd3d6e580bc88b9a160066d9e1177fe1

I did not have the bug-entry when the changelog was written
so no references there.

	Sam

Re: [Bug #14278] New message "NOHZ: local_softirq_pending 08" at each ping request

[Rafael J. Wysocki]

On Tuesday 13 October 2009, John W. Linville wrote:
> On Mon, Oct 12, 2009 at 11:49:39PM +0200, Rafael J. Wysocki wrote:
> > On Monday 12 October 2009, Michael Buesch wrote:
> > > On Monday 12 October 2009 00:22:00 Rafael J. Wysocki wrote:
> > > > This message has been generated automatically as a part of a report
> > > > of recent regressions.
> > > > 
> > > > The following bug entry is on the current list of known regressions
> > > > from 2.6.31.  Please verify if it still should be listed and let me know
> > > > (either way).
> > > > 
> > > > 
> > > > Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14278
> > > > Subject		: New message "NOHZ: local_softirq_pending 08" at each ping request
> > > > Submitter	: Christian Casteyde <casteyde.christian-GANU6spQydw@public.gmane.org>
> > > > Date		: 2009-09-30 18:12 (12 days old)
> > > > Handled-By	: Michael Buesch <mb-fseUSCV1ubazQB+pC5nmwQ@public.gmane.org>
> > > 
> > > I think John or Johannes is handling it, as my patch was rejected.
> > > 
> > > > Patch		: http://bugzilla.kernel.org/attachment.cgi?id=23220
> > 
> > Johannes, what's the status here?
> 
> Johannes posted a patch here:
> 
> 	http://marc.info/?l=linux-wireless&m=125525657120789&w=2
> 
> It is on my merge train, but it hasn't gone to Linus (or Dave) just
> yet -- probably today.
> 
> Hth!

Yes, it does, I've updated the bug entry with the link to the Johannes' patch.

Thanks,
Rafael

Re: [Bug #14389] Build system issue

[Rafael J. Wysocki]

On Tuesday 13 October 2009, Sam Ravnborg wrote:
> On Mon, Oct 12, 2009 at 12:22:09AM +0200, Rafael J. Wysocki wrote:
> > This message has been generated automatically as a part of a report
> > of recent regressions.
> > 
> > The following bug entry is on the current list of known regressions
> > from 2.6.31.  Please verify if it still should be listed and let me know
> > (either way).
> > 
> > 
> > Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14389
> > Subject		: Build system issue
> > Submitter	: Peter Zijlstra <peterz@infradead.org>
> > Date		: 2009-10-09 8:58 (3 days old)
> > First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=575543347b5baed0ca927cb90ba8807396fe9cc9
> > References	: http://marc.info/?l=linux-kernel&m=125507914909152&w=4
> 
> It is fixed by: 2331d1a6cd3d6e580bc88b9a160066d9e1177fe1
> 
> I did not have the bug-entry when the changelog was written
> so no references there.

Thanks, closing.

Rafael

Re: [Bug #14378] Problems with net/core/skbuff.c

[Massimo Cetra]

Massimo Cetra ha scritto:
> Eric Dumazet ha scritto:
>> Could you please following patch ?
>>
>> diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
>> index 8d00976..54bf091 100644
>> --- a/drivers/net/virtio_net.c
>> +++ b/drivers/net/virtio_net.c
>> @@ -454,7 +454,7 @@ static unsigned int free_old_xmit_skbs(struct 
>> virtnet_info *vi)
>>          vi->dev->stats.tx_bytes += skb->len;
>>          vi->dev->stats.tx_packets++;
>>          tot_sgs += skb_vnet_hdr(skb)->num_sg;
>> -        kfree_skb(skb);
>> +        dev_kfree_skb_any(skb);
>>      }
>>      return tot_sgs;
>>  }
>>
>>
>>   
> Thank you very much.
>
> Compiling.
> It' will be in production in a few minutes.
> I'll let you know if the problem arises again.
> give me a couple of days because this problem is randomly triggered.
> Sometimes it happens multiple times a day, sometimes none.
>
Eric,
thanks for the patch.
The problem didn't arise again and i haven't seen any warning like that 
on both servers where that problem was happening more frequently.

I would say that it's fixed and if it's not, i'll let you know as soon 
as it happens again.

Thanks again,
Max

[PATCH] virtio_net: use dev_kfree_skb_any() in free_old_xmit_skbs()

[Eric Dumazet]

Massimo Cetra a écrit :
> Eric,
> thanks for the patch.
> The problem didn't arise again and i haven't seen any warning like that
> on both servers where that problem was happening more frequently.
> 
> I would say that it's fixed and if it's not, i'll let you know as soon
> as it happens again.
> 

Thanks Massimo, I think patch is reasonably safe and should be taken as is :


[PATCH] virtio_net: use dev_kfree_skb_any() in free_old_xmit_skbs()

Because netpoll can call netdevice start_xmit() method with
irqs disabled, drivers should not call kfree_skb() from
their start_xmit(), but use dev_kfree_skb_any() instead.

Oct  8 11:16:52 172.30.1.31 [113074.791813] ------------[ cut here ]------------
Oct  8 11:16:52 172.30.1.31 [113074.791813] WARNING: at net/core/skbuff.c:398 \
                skb_release_head_state+0x64/0xc8()
Oct  8 11:16:52 172.30.1.31 [113074.791813] Hardware name: 
Oct  8 11:16:52 172.30.1.31 [113074.791813] Modules linked in: netconsole ocfs2 jbd2 quota_tree \
ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue configfs crc32c drbd cn loop \
serio_raw psmouse snd_pcm snd_timer snd soundcore snd_page_alloc virtio_net pcspkr parport_pc parport \
i2c_piix4 i2c_core button processor evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot \
dm_mod ide_cd_mod cdrom ata_generic ata_piix virtio_blk libata scsi_mod piix ide_pci_generic ide_core \
                virtio_pci virtio_ring virtio floppy thermal fan thermal_sys [last unloaded: netconsole]
Oct  8 11:16:52 172.30.1.31 [113074.791813] Pid: 11132, comm: php5-cgi Tainted: G        W  \
                2.6.31.2-vserver #1
Oct  8 11:16:52 172.30.1.31 [113074.791813] Call Trace:
Oct  8 11:16:52 172.30.1.31 [113074.791813] <IRQ>  [<ffffffff81253cd5>] ? \
                skb_release_head_state+0x64/0xc8
Oct  8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
Oct  8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3
Oct  8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
Oct  8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d
Oct  8 11:16:52 172.30.1.31 [113074.791813] [<ffffffffa01cb139>] ? free_old_xmit_skbs+0x51/0x6e \
                [virtio_net]
Oct  8 11:16:52 172.30.1.31 [113074.791813] [<ffffffffa01cbc85>] ? start_xmit+0x26/0xf2 [virtio_net]
Oct  8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205
Oct  8 11:16:52 172.30.1.31 [113074.791813] [<ffffffffa0429216>] ? write_msg+0x90/0xeb [netconsole]
Oct  8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f
Oct  8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
Oct  8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba
Oct  8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b
Oct  8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff8106b142>] ? vx_update_load+0x18/0x13e
Oct  8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff81308309>] ? printk+0x4e/0x5d
Oct  8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
Oct  8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf
Oct  8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49
Oct  8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff810626b7>] ? ktime_get+0xc/0x41
Oct  8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146
Oct  8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93
Oct  8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20
Oct  8 11:16:52 172.30.1.31 [113074.791813] <EOI>  [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31

Reported-and-tested-by: Massimo Cetra <mcetra-BBpJ+9iBSNKonA0d6jMUrA@public.gmane.org>
Signed-off-by: Eric Dumazet <eric.dumazet-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
Bug-Entry: http://bugzilla.kernel.org/show_bug.cgi?id=14378
---

diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index 8d00976..54bf091 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c
@@ -454,7 +454,7 @@ static unsigned int free_old_xmit_skbs(struct virtnet_info *vi)
 		vi->dev->stats.tx_bytes += skb->len;
 		vi->dev->stats.tx_packets++;
 		tot_sgs += skb_vnet_hdr(skb)->num_sg;
-		kfree_skb(skb);
+		dev_kfree_skb_any(skb);
 	}
 	return tot_sgs;
 }

Re: [PATCH] virtio_net: use dev_kfree_skb_any() in free_old_xmit_skbs()

[David Miller]

From: Eric Dumazet <eric.dumazet@gmail.com>
Date: Thu, 15 Oct 2009 02:36:43 +0200

> Massimo Cetra a écrit :
>> Eric,
>> thanks for the patch.
>> The problem didn't arise again and i haven't seen any warning like that
>> on both servers where that problem was happening more frequently.
>> 
>> I would say that it's fixed and if it's not, i'll let you know as soon
>> as it happens again.
>> 
> 
> Thanks Massimo, I think patch is reasonably safe and should be taken as is :
> 
> 
> [PATCH] virtio_net: use dev_kfree_skb_any() in free_old_xmit_skbs()

Applied, thanks Eric.