From: Brad Campbell <lists2009@fnarfbargle.com>
To: Eric Dumazet <eric.dumazet@gmail.com>
Cc: Avi Kivity <avi@redhat.com>, CaT <cat@zip.com.au>,
Borislav Petkov <bp@alien8.de>,
linux-kernel@vger.kernel.org, kvm@vger.kernel.org,
netdev <netdev@vger.kernel.org>
Subject: Re: KVM induced panic on 2.6.38[2367] & 2.6.39
Date: Sat, 20 Aug 2011 21:16:39 +0800 [thread overview]
Message-ID: <4E4FB3B7.4040501@fnarfbargle.com> (raw)
In-Reply-To: <1307453874.3091.14.camel@edumazet-laptop>
On 07/06/11 21:37, Eric Dumazet wrote:
> Le mardi 07 juin 2011 à 21:27 +0800, Brad Campbell a écrit :
>> On 07/06/11 04:22, Eric Dumazet wrote:
>>
>>> Could you please try latest linux-2.6 tree ?
>>>
>>> We fixed many networking bugs that could explain your crash.
>>>
>>>
>>>
>>>
>>
>> No good I'm afraid.
>>
>> [ 543.040056]
>> =============================================================================
>> [ 543.040136] BUG ip_dst_cache: Padding overwritten.
>> 0xffff8803e4217ffe-0xffff8803e4217fff
>> [ 543.040194]
>
> Thats pretty strange : These are the last two bytes of a page, set to
> 0x0000 (a 16 bit value)
>
> There is no way a dst field could actually sit on this location (its a
> padding), since a dst is a bit less than 256 bytes (0xe8), and each
> entry is aligned on a 64byte address.
>
> grep dst /proc/slabinfo
>
> ip_dst_cache 32823 62944 256 32 2 : tunables 0 0
> 0 : slabdata 1967 1967 0
>
> sizeof(struct rtable)=0xe8
>
>
>> -----------------------------------------------------------------------------
>> [ 543.040198]
>> [ 543.040298] INFO: Slab 0xffffea000d9e74d0 objects=25 used=25 fp=0x
>> (null) flags=0x8000000000004081
>> [ 543.040364] Pid: 4576, comm: kworker/1:2 Not tainted 3.0.0-rc2 #1
>> [ 543.040415] Call Trace:
>> [ 543.040472] [<ffffffff810b9c1d>] ? slab_err+0xad/0xd0
>> [ 543.040528] [<ffffffff8102e034>] ? check_preempt_wakeup+0xa4/0x160
>> [ 543.040595] [<ffffffff810ba206>] ? slab_pad_check+0x126/0x170
>> [ 543.040650] [<ffffffff8133045b>] ? dst_destroy+0x8b/0x110
>> [ 543.040701] [<ffffffff810ba29a>] ? check_slab+0x4a/0xc0
>> [ 543.040753] [<ffffffff810baf2d>] ? free_debug_processing+0x2d/0x250
>> [ 543.040808] [<ffffffff810bb27b>] ? __slab_free+0x12b/0x140
>> [ 543.040862] [<ffffffff810bbe99>] ? kmem_cache_free+0x99/0xa0
>> [ 543.040915] [<ffffffff8133045b>] ? dst_destroy+0x8b/0x110
>> [ 543.040967] [<ffffffff813307f6>] ? dst_gc_task+0x196/0x1f0
>> [ 543.041021] [<ffffffff8104e954>] ? queue_delayed_work_on+0x154/0x160
>> [ 543.041081] [<ffffffff813066fe>] ? do_dbs_timer+0x20e/0x3d0
>> [ 543.041133] [<ffffffff81330660>] ? dst_alloc+0x180/0x180
>> [ 543.041187] [<ffffffff8104f28b>] ? process_one_work+0xfb/0x3b0
>> [ 543.041242] [<ffffffff8104f964>] ? worker_thread+0x144/0x3d0
>> [ 543.041296] [<ffffffff8102cc10>] ? __wake_up_common+0x50/0x80
>> [ 543.041678] [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
>> [ 543.041729] [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
>> [ 543.041782] [<ffffffff81053436>] ? kthread+0x96/0xa0
>> [ 543.041835] [<ffffffff813e1d14>] ? kernel_thread_helper+0x4/0x10
>> [ 543.041890] [<ffffffff810533a0>] ? kthread_worker_fn+0x120/0x120
>> [ 543.041944] [<ffffffff813e1d10>] ? gs_change+0xb/0xb
>> [ 543.041993] Padding 0xffff8803e4217f40: 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [ 543.042718] Padding 0xffff8803e4217f50: 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [ 543.043433] Padding 0xffff8803e4217f60: 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [ 543.044155] Padding 0xffff8803e4217f70: 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [ 543.044866] Padding 0xffff8803e4217f80: 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [ 543.045590] Padding 0xffff8803e4217f90: 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [ 543.046311] Padding 0xffff8803e4217fa0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [ 543.047034] Padding 0xffff8803e4217fb0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [ 543.047755] Padding 0xffff8803e4217fc0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [ 543.048474] Padding 0xffff8803e4217fd0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [ 543.049203] Padding 0xffff8803e4217fe0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [ 543.049909] Padding 0xffff8803e4217ff0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 00 00 ZZZZZZZZZZZZZZ..
>> [ 543.050021] FIX ip_dst_cache: Restoring
>> 0xffff8803e4217f40-0xffff8803e4217fff=0x5a
>> [ 543.050021]
>>
>> Dropped -mm, Hugh and Andrea from CC as this does not appear to be mm or
>> ksm related.
>>
>> I'll pare down the firewall and see if I can make it break easier with a
>> smaller test set.
>
> Hmm, not sure now :(
>
> Could you reproduce another bug please ?
I know this is an old one, but I recently purchased a second system to
allow me to test and bisect this off-line (the live system is too much
of a headache to bisect on).
brad@test:/raid10/src/linux-2.6$ git bisect log
git bisect start
# good: [9fe6206f400646a2322096b56c59891d530e8d51] Linux 2.6.35
git bisect good 9fe6206f400646a2322096b56c59891d530e8d51
# bad: [da5cabf80e2433131bf0ed8993abc0f7ea618c73] Linux 2.6.36-rc1
git bisect bad da5cabf80e2433131bf0ed8993abc0f7ea618c73
# bad: [0f477dd0851bdcee82923da66a7fc4a44cb1bc3d] Merge branch
'x86-cpu-for-linus' of
git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
git bisect bad 0f477dd0851bdcee82923da66a7fc4a44cb1bc3d
# bad: [3ff1c25927e3af61c6bf0e4ed959504058ae4565] phy/marvell: add
88ec048 support
git bisect bad 3ff1c25927e3af61c6bf0e4ed959504058ae4565
# good: [05318bc905467237d4aa68a701f6e92a2b332218] Merge branch 'master'
of git://git.kernel.org/pub/scm/linux/kernel/git/linville/wireless-next-2.6
git bisect good 05318bc905467237d4aa68a701f6e92a2b332218
# bad: [2ba13ed678775195e8255b4e503c59d48b615bd8] Bluetooth: Remove
check for supported mode
git bisect bad 2ba13ed678775195e8255b4e503c59d48b615bd8
# bad: [1e2cfeef060fa0270f9a2d66b1218c12c05062e0] Revert "tc35815: fix
iomap leak"
git bisect bad 1e2cfeef060fa0270f9a2d66b1218c12c05062e0
# bad: [d9bed6bbd4f2a0120c93fed68605950651e1f225] isdn/gigaset: remove
EXPERIMENTAL tag from GIGASET_CAPI
git bisect bad d9bed6bbd4f2a0120c93fed68605950651e1f225
# bad: [d117b6665847084cfe8a44b870f771153e18991d] fealnx: Use the
instance of net_device_stats from net_device.
git bisect bad d117b6665847084cfe8a44b870f771153e18991d
# bad: [e490c1defec4236a6a131fe2d13bf7ba787c02f8] Merge branch 'master'
of git://git.kernel.org/pub/scm/linux/kernel/git/kaber/nf-next-2.6
git bisect bad e490c1defec4236a6a131fe2d13bf7ba787c02f8
# bad: [0a17d8c744e44617a3c22e7af68b4c5c9c1c5dba] ixgbe: use NETIF_F_LRO
git bisect bad 0a17d8c744e44617a3c22e7af68b4c5c9c1c5dba
# bad: [ede3ef0d940ef052466f42c849390b23c6859abc] igb: fix PHY config
access on 82580
git bisect bad ede3ef0d940ef052466f42c849390b23c6859abc
# good: [ee3cb6295144b0adfa75ccaca307643a6998b1e2] be2net: changes to
properly provide phy details
git bisect good ee3cb6295144b0adfa75ccaca307643a6998b1e2
# bad: [7475271004b66e9c22e1bb28f240a38c5d6fe76e] x86: Drop
CONFIG_MCORE2 check around setting of NET_IP_ALIGN
git bisect bad 7475271004b66e9c22e1bb28f240a38c5d6fe76e
brad@test:/raid10/src/linux-2.6$ git bisect log
git bisect start
# good: [9fe6206f400646a2322096b56c59891d530e8d51] Linux 2.6.35
git bisect good 9fe6206f400646a2322096b56c59891d530e8d51
# bad: [da5cabf80e2433131bf0ed8993abc0f7ea618c73] Linux 2.6.36-rc1
git bisect bad da5cabf80e2433131bf0ed8993abc0f7ea618c73
# bad: [0f477dd0851bdcee82923da66a7fc4a44cb1bc3d] Merge branch
'x86-cpu-for-linus' of
git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
git bisect bad 0f477dd0851bdcee82923da66a7fc4a44cb1bc3d
# bad: [3ff1c25927e3af61c6bf0e4ed959504058ae4565] phy/marvell: add
88ec048 support
git bisect bad 3ff1c25927e3af61c6bf0e4ed959504058ae4565
# good: [05318bc905467237d4aa68a701f6e92a2b332218] Merge branch 'master'
of git://git.kernel.org/pub/scm/linux/kernel/git/linville/wireless-next-2.6
git bisect good 05318bc905467237d4aa68a701f6e92a2b332218
# bad: [2ba13ed678775195e8255b4e503c59d48b615bd8] Bluetooth: Remove
check for supported mode
git bisect bad 2ba13ed678775195e8255b4e503c59d48b615bd8
# bad: [1e2cfeef060fa0270f9a2d66b1218c12c05062e0] Revert "tc35815: fix
iomap leak"
git bisect bad 1e2cfeef060fa0270f9a2d66b1218c12c05062e0
# bad: [d9bed6bbd4f2a0120c93fed68605950651e1f225] isdn/gigaset: remove
EXPERIMENTAL tag from GIGASET_CAPI
git bisect bad d9bed6bbd4f2a0120c93fed68605950651e1f225
# bad: [d117b6665847084cfe8a44b870f771153e18991d] fealnx: Use the
instance of net_device_stats from net_device.
git bisect bad d117b6665847084cfe8a44b870f771153e18991d
# bad: [e490c1defec4236a6a131fe2d13bf7ba787c02f8] Merge branch 'master'
of git://git.kernel.org/pub/scm/linux/kernel/git/kaber/nf-next-2.6
git bisect bad e490c1defec4236a6a131fe2d13bf7ba787c02f8
# bad: [0a17d8c744e44617a3c22e7af68b4c5c9c1c5dba] ixgbe: use NETIF_F_LRO
git bisect bad 0a17d8c744e44617a3c22e7af68b4c5c9c1c5dba
# bad: [ede3ef0d940ef052466f42c849390b23c6859abc] igb: fix PHY config
access on 82580
git bisect bad ede3ef0d940ef052466f42c849390b23c6859abc
# good: [ee3cb6295144b0adfa75ccaca307643a6998b1e2] be2net: changes to
properly provide phy details
git bisect good ee3cb6295144b0adfa75ccaca307643a6998b1e2
# bad: [7475271004b66e9c22e1bb28f240a38c5d6fe76e] x86: Drop
CONFIG_MCORE2 check around setting of NET_IP_ALIGN
git bisect bad 7475271004b66e9c22e1bb28f240a38c5d6fe76e
brad@test:/raid10/src/linux-2.6$ git bisect good
7475271004b66e9c22e1bb28f240a38c5d6fe76e is the first bad commit
commit 7475271004b66e9c22e1bb28f240a38c5d6fe76e
Author: Alexander Duyck <alexander.h.duyck@intel.com>
Date: Thu Jul 1 13:28:27 2010 +0000
x86: Drop CONFIG_MCORE2 check around setting of NET_IP_ALIGN
This patch removes the CONFIG_MCORE2 check from around
NET_IP_ALIGN. It is
based on a suggestion from Andi Kleen. The assumption is that
there are
not any x86 cores where unaligned access is really slow, and this
change
would allow for a performance improvement to still exist on
configurations
that are not necessarily optimized for Core 2.
Cc: Andi Kleen <ak@linux.intel.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: x86@kernel.org
Signed-off-by: Alexander Duyck <alexander.h.duyck@intel.com>
Signed-off-by: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
Acked-by: H. Peter Anvin <hpa@zytor.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
:040000 040000 5a15867789080a2f67a74b17c4422f85b7a9fb4a
b98769348bd765731ca3ff03b33764257e23226c M arch
I can confirm this bug exists in the 3.0 kernel, however I'm unable to
reproduce it on todays git.
So anyone using netfilter, kvm and bridge on kernels between 2.6.36-rc1
and 3.0 may hit this bug, but it looks like it is fixed in the current
3.1-rc kernels.
next prev parent reply other threads:[~2011-08-20 13:54 UTC|newest]
Thread overview: 54+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-05-31 1:24 KVM induced panic on 2.6.38[2367] & 2.6.39 Brad Campbell
2011-05-31 5:47 ` Borislav Petkov
2011-05-31 9:26 ` Brad Campbell
2011-05-31 10:38 ` Borislav Petkov
2011-05-31 14:24 ` Brad Campbell
2011-05-31 22:31 ` Hugh Dickins
2011-06-01 0:18 ` Brad Campbell
2011-06-01 0:37 ` Brad Campbell
2011-06-01 1:15 ` Andrea Arcangeli
2011-06-01 2:03 ` Brad Campbell
2011-06-01 4:52 ` Hugh Dickins
2011-06-01 6:31 ` Brad Campbell
2011-06-01 6:56 ` Avi Kivity
2011-06-01 9:29 ` Brad Campbell
2011-06-01 9:40 ` Avi Kivity
2011-06-01 9:41 ` Avi Kivity
2011-06-01 10:53 ` Brad Campbell
2011-06-01 11:09 ` Avi Kivity
2011-06-01 11:18 ` CaT
2011-06-01 11:52 ` Brad Campbell
2011-06-01 23:03 ` CaT
2011-06-03 13:38 ` Brad Campbell
2011-06-03 15:50 ` Bernhard Held
2011-06-03 16:07 ` Brad Campbell
2011-06-06 20:10 ` Bart De Schuymer
2011-06-06 20:23 ` Eric Dumazet
2011-06-07 3:33 ` Brad Campbell
2011-06-07 13:30 ` Patrick McHardy
2011-06-07 14:40 ` Brad Campbell
2011-06-07 15:35 ` Patrick McHardy
2011-06-07 18:31 ` Eric Dumazet
2011-06-07 22:57 ` Patrick McHardy
2011-06-08 0:18 ` Brad Campbell
2011-06-08 3:59 ` Eric Dumazet
2011-06-08 17:02 ` Brad Campbell
2011-06-08 21:22 ` Eric Dumazet
2011-06-10 2:52 ` Simon Horman
2011-06-10 12:37 ` Mark Lord
2011-06-10 16:43 ` Henrique de Moraes Holschuh
2011-06-12 15:38 ` Avi Kivity
2011-06-07 23:43 ` Brad Campbell
2011-06-07 18:04 ` Bart De Schuymer
2011-06-08 0:15 ` Brad Campbell
2011-06-05 8:14 ` Avi Kivity
2011-06-05 13:45 ` Brad Campbell
2011-06-05 13:58 ` Avi Kivity
2011-06-06 20:22 ` Eric Dumazet
2011-06-07 13:27 ` Brad Campbell
2011-06-07 13:37 ` Eric Dumazet
2011-06-07 15:15 ` Brad Campbell
2011-08-20 13:16 ` Brad Campbell [this message]
2011-08-22 6:36 ` Avi Kivity
2011-08-22 6:45 ` Eric Dumazet
2011-08-22 11:45 ` Brad Campbell
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4E4FB3B7.4040501@fnarfbargle.com \
--to=lists2009@fnarfbargle.com \
--cc=avi@redhat.com \
--cc=bp@alien8.de \
--cc=cat@zip.com.au \
--cc=eric.dumazet@gmail.com \
--cc=kvm@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=netdev@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox