From: Brad Campbell <lists2009@fnarfbargle.com>
To: Eric Dumazet <eric.dumazet@gmail.com>
Cc: Avi Kivity <avi@redhat.com>, CaT <cat@zip.com.au>,
Borislav Petkov <bp@alien8.de>,
linux-kernel@vger.kernel.org, kvm@vger.kernel.org,
netdev <netdev@vger.kernel.org>
Subject: Re: KVM induced panic on 2.6.38[2367] & 2.6.39
Date: Sat, 20 Aug 2011 21:16:39 +0800 [thread overview]
Message-ID: <4E4FB3B7.4040501@fnarfbargle.com> (raw)
In-Reply-To: <1307453874.3091.14.camel@edumazet-laptop>
On 07/06/11 21:37, Eric Dumazet wrote:
> Le mardi 07 juin 2011 à 21:27 +0800, Brad Campbell a écrit :
>> On 07/06/11 04:22, Eric Dumazet wrote:
>>
>>> Could you please try latest linux-2.6 tree ?
>>>
>>> We fixed many networking bugs that could explain your crash.
>>>
>>>
>>>
>>>
>>
>> No good I'm afraid.
>>
>> [ 543.040056]
>> =============================================================================
>> [ 543.040136] BUG ip_dst_cache: Padding overwritten.
>> 0xffff8803e4217ffe-0xffff8803e4217fff
>> [ 543.040194]
>
> Thats pretty strange : These are the last two bytes of a page, set to
> 0x0000 (a 16 bit value)
>
> There is no way a dst field could actually sit on this location (its a
> padding), since a dst is a bit less than 256 bytes (0xe8), and each
> entry is aligned on a 64byte address.
>
> grep dst /proc/slabinfo
>
> ip_dst_cache 32823 62944 256 32 2 : tunables 0 0
> 0 : slabdata 1967 1967 0
>
> sizeof(struct rtable)=0xe8
>
>
>> -----------------------------------------------------------------------------
>> [ 543.040198]
>> [ 543.040298] INFO: Slab 0xffffea000d9e74d0 objects=25 used=25 fp=0x
>> (null) flags=0x8000000000004081
>> [ 543.040364] Pid: 4576, comm: kworker/1:2 Not tainted 3.0.0-rc2 #1
>> [ 543.040415] Call Trace:
>> [ 543.040472] [<ffffffff810b9c1d>] ? slab_err+0xad/0xd0
>> [ 543.040528] [<ffffffff8102e034>] ? check_preempt_wakeup+0xa4/0x160
>> [ 543.040595] [<ffffffff810ba206>] ? slab_pad_check+0x126/0x170
>> [ 543.040650] [<ffffffff8133045b>] ? dst_destroy+0x8b/0x110
>> [ 543.040701] [<ffffffff810ba29a>] ? check_slab+0x4a/0xc0
>> [ 543.040753] [<ffffffff810baf2d>] ? free_debug_processing+0x2d/0x250
>> [ 543.040808] [<ffffffff810bb27b>] ? __slab_free+0x12b/0x140
>> [ 543.040862] [<ffffffff810bbe99>] ? kmem_cache_free+0x99/0xa0
>> [ 543.040915] [<ffffffff8133045b>] ? dst_destroy+0x8b/0x110
>> [ 543.040967] [<ffffffff813307f6>] ? dst_gc_task+0x196/0x1f0
>> [ 543.041021] [<ffffffff8104e954>] ? queue_delayed_work_on+0x154/0x160
>> [ 543.041081] [<ffffffff813066fe>] ? do_dbs_timer+0x20e/0x3d0
>> [ 543.041133] [<ffffffff81330660>] ? dst_alloc+0x180/0x180
>> [ 543.041187] [<ffffffff8104f28b>] ? process_one_work+0xfb/0x3b0
>> [ 543.041242] [<ffffffff8104f964>] ? worker_thread+0x144/0x3d0
>> [ 543.041296] [<ffffffff8102cc10>] ? __wake_up_common+0x50/0x80
>> [ 543.041678] [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
>> [ 543.041729] [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
>> [ 543.041782] [<ffffffff81053436>] ? kthread+0x96/0xa0
>> [ 543.041835] [<ffffffff813e1d14>] ? kernel_thread_helper+0x4/0x10
>> [ 543.041890] [<ffffffff810533a0>] ? kthread_worker_fn+0x120/0x120
>> [ 543.041944] [<ffffffff813e1d10>] ? gs_change+0xb/0xb
>> [ 543.041993] Padding 0xffff8803e4217f40: 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [ 543.042718] Padding 0xffff8803e4217f50: 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [ 543.043433] Padding 0xffff8803e4217f60: 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [ 543.044155] Padding 0xffff8803e4217f70: 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [ 543.044866] Padding 0xffff8803e4217f80: 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [ 543.045590] Padding 0xffff8803e4217f90: 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [ 543.046311] Padding 0xffff8803e4217fa0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [ 543.047034] Padding 0xffff8803e4217fb0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [ 543.047755] Padding 0xffff8803e4217fc0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [ 543.048474] Padding 0xffff8803e4217fd0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [ 543.049203] Padding 0xffff8803e4217fe0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [ 543.049909] Padding 0xffff8803e4217ff0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 00 00 ZZZZZZZZZZZZZZ..
>> [ 543.050021] FIX ip_dst_cache: Restoring
>> 0xffff8803e4217f40-0xffff8803e4217fff=0x5a
>> [ 543.050021]
>>
>> Dropped -mm, Hugh and Andrea from CC as this does not appear to be mm or
>> ksm related.
>>
>> I'll pare down the firewall and see if I can make it break easier with a
>> smaller test set.
>
> Hmm, not sure now :(
>
> Could you reproduce another bug please ?
I know this is an old one, but I recently purchased a second system to
allow me to test and bisect this off-line (the live system is too much
of a headache to bisect on).
brad@test:/raid10/src/linux-2.6$ git bisect log
git bisect start
# good: [9fe6206f400646a2322096b56c59891d530e8d51] Linux 2.6.35
git bisect good 9fe6206f400646a2322096b56c59891d530e8d51
# bad: [da5cabf80e2433131bf0ed8993abc0f7ea618c73] Linux 2.6.36-rc1
git bisect bad da5cabf80e2433131bf0ed8993abc0f7ea618c73
# bad: [0f477dd0851bdcee82923da66a7fc4a44cb1bc3d] Merge branch
'x86-cpu-for-linus' of
git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
git bisect bad 0f477dd0851bdcee82923da66a7fc4a44cb1bc3d
# bad: [3ff1c25927e3af61c6bf0e4ed959504058ae4565] phy/marvell: add
88ec048 support
git bisect bad 3ff1c25927e3af61c6bf0e4ed959504058ae4565
# good: [05318bc905467237d4aa68a701f6e92a2b332218] Merge branch 'master'
of git://git.kernel.org/pub/scm/linux/kernel/git/linville/wireless-next-2.6
git bisect good 05318bc905467237d4aa68a701f6e92a2b332218
# bad: [2ba13ed678775195e8255b4e503c59d48b615bd8] Bluetooth: Remove
check for supported mode
git bisect bad 2ba13ed678775195e8255b4e503c59d48b615bd8
# bad: [1e2cfeef060fa0270f9a2d66b1218c12c05062e0] Revert "tc35815: fix
iomap leak"
git bisect bad 1e2cfeef060fa0270f9a2d66b1218c12c05062e0
# bad: [d9bed6bbd4f2a0120c93fed68605950651e1f225] isdn/gigaset: remove
EXPERIMENTAL tag from GIGASET_CAPI
git bisect bad d9bed6bbd4f2a0120c93fed68605950651e1f225
# bad: [d117b6665847084cfe8a44b870f771153e18991d] fealnx: Use the
instance of net_device_stats from net_device.
git bisect bad d117b6665847084cfe8a44b870f771153e18991d
# bad: [e490c1defec4236a6a131fe2d13bf7ba787c02f8] Merge branch 'master'
of git://git.kernel.org/pub/scm/linux/kernel/git/kaber/nf-next-2.6
git bisect bad e490c1defec4236a6a131fe2d13bf7ba787c02f8
# bad: [0a17d8c744e44617a3c22e7af68b4c5c9c1c5dba] ixgbe: use NETIF_F_LRO
git bisect bad 0a17d8c744e44617a3c22e7af68b4c5c9c1c5dba
# bad: [ede3ef0d940ef052466f42c849390b23c6859abc] igb: fix PHY config
access on 82580
git bisect bad ede3ef0d940ef052466f42c849390b23c6859abc
# good: [ee3cb6295144b0adfa75ccaca307643a6998b1e2] be2net: changes to
properly provide phy details
git bisect good ee3cb6295144b0adfa75ccaca307643a6998b1e2
# bad: [7475271004b66e9c22e1bb28f240a38c5d6fe76e] x86: Drop
CONFIG_MCORE2 check around setting of NET_IP_ALIGN
git bisect bad 7475271004b66e9c22e1bb28f240a38c5d6fe76e
brad@test:/raid10/src/linux-2.6$ git bisect log
git bisect start
# good: [9fe6206f400646a2322096b56c59891d530e8d51] Linux 2.6.35
git bisect good 9fe6206f400646a2322096b56c59891d530e8d51
# bad: [da5cabf80e2433131bf0ed8993abc0f7ea618c73] Linux 2.6.36-rc1
git bisect bad da5cabf80e2433131bf0ed8993abc0f7ea618c73
# bad: [0f477dd0851bdcee82923da66a7fc4a44cb1bc3d] Merge branch
'x86-cpu-for-linus' of
git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
git bisect bad 0f477dd0851bdcee82923da66a7fc4a44cb1bc3d
# bad: [3ff1c25927e3af61c6bf0e4ed959504058ae4565] phy/marvell: add
88ec048 support
git bisect bad 3ff1c25927e3af61c6bf0e4ed959504058ae4565
# good: [05318bc905467237d4aa68a701f6e92a2b332218] Merge branch 'master'
of git://git.kernel.org/pub/scm/linux/kernel/git/linville/wireless-next-2.6
git bisect good 05318bc905467237d4aa68a701f6e92a2b332218
# bad: [2ba13ed678775195e8255b4e503c59d48b615bd8] Bluetooth: Remove
check for supported mode
git bisect bad 2ba13ed678775195e8255b4e503c59d48b615bd8
# bad: [1e2cfeef060fa0270f9a2d66b1218c12c05062e0] Revert "tc35815: fix
iomap leak"
git bisect bad 1e2cfeef060fa0270f9a2d66b1218c12c05062e0
# bad: [d9bed6bbd4f2a0120c93fed68605950651e1f225] isdn/gigaset: remove
EXPERIMENTAL tag from GIGASET_CAPI
git bisect bad d9bed6bbd4f2a0120c93fed68605950651e1f225
# bad: [d117b6665847084cfe8a44b870f771153e18991d] fealnx: Use the
instance of net_device_stats from net_device.
git bisect bad d117b6665847084cfe8a44b870f771153e18991d
# bad: [e490c1defec4236a6a131fe2d13bf7ba787c02f8] Merge branch 'master'
of git://git.kernel.org/pub/scm/linux/kernel/git/kaber/nf-next-2.6
git bisect bad e490c1defec4236a6a131fe2d13bf7ba787c02f8
# bad: [0a17d8c744e44617a3c22e7af68b4c5c9c1c5dba] ixgbe: use NETIF_F_LRO
git bisect bad 0a17d8c744e44617a3c22e7af68b4c5c9c1c5dba
# bad: [ede3ef0d940ef052466f42c849390b23c6859abc] igb: fix PHY config
access on 82580
git bisect bad ede3ef0d940ef052466f42c849390b23c6859abc
# good: [ee3cb6295144b0adfa75ccaca307643a6998b1e2] be2net: changes to
properly provide phy details
git bisect good ee3cb6295144b0adfa75ccaca307643a6998b1e2
# bad: [7475271004b66e9c22e1bb28f240a38c5d6fe76e] x86: Drop
CONFIG_MCORE2 check around setting of NET_IP_ALIGN
git bisect bad 7475271004b66e9c22e1bb28f240a38c5d6fe76e
brad@test:/raid10/src/linux-2.6$ git bisect good
7475271004b66e9c22e1bb28f240a38c5d6fe76e is the first bad commit
commit 7475271004b66e9c22e1bb28f240a38c5d6fe76e
Author: Alexander Duyck <alexander.h.duyck@intel.com>
Date: Thu Jul 1 13:28:27 2010 +0000
x86: Drop CONFIG_MCORE2 check around setting of NET_IP_ALIGN
This patch removes the CONFIG_MCORE2 check from around
NET_IP_ALIGN. It is
based on a suggestion from Andi Kleen. The assumption is that
there are
not any x86 cores where unaligned access is really slow, and this
change
would allow for a performance improvement to still exist on
configurations
that are not necessarily optimized for Core 2.
Cc: Andi Kleen <ak@linux.intel.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: x86@kernel.org
Signed-off-by: Alexander Duyck <alexander.h.duyck@intel.com>
Signed-off-by: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
Acked-by: H. Peter Anvin <hpa@zytor.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
:040000 040000 5a15867789080a2f67a74b17c4422f85b7a9fb4a
b98769348bd765731ca3ff03b33764257e23226c M arch
I can confirm this bug exists in the 3.0 kernel, however I'm unable to
reproduce it on todays git.
So anyone using netfilter, kvm and bridge on kernels between 2.6.36-rc1
and 3.0 may hit this bug, but it looks like it is fixed in the current
3.1-rc kernels.
next prev parent reply other threads:[~2011-08-20 13:54 UTC|newest]
Thread overview: 111+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-05-31 1:24 KVM induced panic on 2.6.38[2367] & 2.6.39 Brad Campbell
2011-05-31 5:47 ` Borislav Petkov
2011-05-31 5:47 ` Borislav Petkov
2011-05-31 9:26 ` Brad Campbell
2011-05-31 9:26 ` Brad Campbell
2011-05-31 10:38 ` Borislav Petkov
2011-05-31 10:38 ` Borislav Petkov
2011-05-31 14:24 ` Brad Campbell
2011-05-31 14:24 ` Brad Campbell
2011-05-31 14:24 ` Brad Campbell
2011-05-31 22:31 ` Hugh Dickins
2011-05-31 22:31 ` Hugh Dickins
2011-06-01 0:18 ` Brad Campbell
2011-06-01 0:18 ` Brad Campbell
2011-06-01 0:37 ` Brad Campbell
2011-06-01 0:37 ` Brad Campbell
2011-06-01 1:15 ` Andrea Arcangeli
2011-06-01 1:15 ` Andrea Arcangeli
2011-06-01 2:03 ` Brad Campbell
2011-06-01 2:03 ` Brad Campbell
2011-06-01 4:52 ` Hugh Dickins
2011-06-01 4:52 ` Hugh Dickins
2011-06-01 6:31 ` Brad Campbell
2011-06-01 6:31 ` Brad Campbell
2011-06-01 6:56 ` Avi Kivity
2011-06-01 6:56 ` Avi Kivity
2011-06-01 9:29 ` Brad Campbell
2011-06-01 9:29 ` Brad Campbell
2011-06-01 9:29 ` Brad Campbell
2011-06-01 9:40 ` Avi Kivity
2011-06-01 9:40 ` Avi Kivity
2011-06-01 9:41 ` Avi Kivity
2011-06-01 9:41 ` Avi Kivity
2011-06-01 10:53 ` Brad Campbell
2011-06-01 10:53 ` Brad Campbell
2011-06-01 11:09 ` Avi Kivity
2011-06-01 11:09 ` Avi Kivity
2011-06-01 11:18 ` CaT
2011-06-01 11:18 ` CaT
2011-06-01 11:52 ` Brad Campbell
2011-06-01 11:52 ` Brad Campbell
2011-06-01 23:03 ` CaT
2011-06-01 23:03 ` CaT
2011-06-03 13:38 ` Brad Campbell
2011-06-03 13:38 ` Brad Campbell
2011-06-03 15:50 ` Bernhard Held
2011-06-03 15:50 ` Bernhard Held
2011-06-03 15:50 ` Bernhard Held
2011-06-03 16:07 ` Brad Campbell
2011-06-03 16:07 ` Brad Campbell
2011-06-06 20:10 ` Bart De Schuymer
2011-06-06 20:10 ` Bart De Schuymer
2011-06-06 20:23 ` Eric Dumazet
2011-06-06 20:23 ` Eric Dumazet
2011-06-06 20:23 ` Eric Dumazet
2011-06-07 3:33 ` Brad Campbell
2011-06-07 3:33 ` Brad Campbell
2011-06-07 13:30 ` Patrick McHardy
2011-06-07 13:30 ` Patrick McHardy
2011-06-07 14:40 ` Brad Campbell
2011-06-07 14:40 ` Brad Campbell
2011-06-07 15:35 ` Patrick McHardy
2011-06-07 15:35 ` Patrick McHardy
2011-06-07 18:31 ` Eric Dumazet
2011-06-07 18:31 ` Eric Dumazet
2011-06-07 18:31 ` Eric Dumazet
2011-06-07 22:57 ` Patrick McHardy
2011-06-07 22:57 ` Patrick McHardy
2011-06-07 22:57 ` Patrick McHardy
2011-06-08 0:18 ` Brad Campbell
2011-06-08 0:18 ` Brad Campbell
2011-06-08 0:18 ` Brad Campbell
2011-06-08 3:59 ` Eric Dumazet
2011-06-08 3:59 ` Eric Dumazet
2011-06-08 3:59 ` Eric Dumazet
2011-06-08 17:02 ` Brad Campbell
2011-06-08 17:02 ` Brad Campbell
2011-06-08 17:02 ` Brad Campbell
2011-06-08 21:22 ` Eric Dumazet
2011-06-08 21:22 ` Eric Dumazet
2011-06-08 21:22 ` Eric Dumazet
2011-06-10 2:52 ` Simon Horman
2011-06-10 2:52 ` Simon Horman
2011-06-10 12:37 ` Mark Lord
2011-06-10 12:37 ` Mark Lord
2011-06-10 16:43 ` Henrique de Moraes Holschuh
2011-06-10 16:43 ` Henrique de Moraes Holschuh
2011-06-12 15:38 ` Avi Kivity
2011-06-12 15:38 ` Avi Kivity
2011-06-07 23:43 ` Brad Campbell
2011-06-07 23:43 ` Brad Campbell
2011-06-07 18:04 ` Bart De Schuymer
2011-06-07 18:04 ` Bart De Schuymer
2011-06-08 0:15 ` Brad Campbell
2011-06-08 0:15 ` Brad Campbell
2011-06-05 8:14 ` Avi Kivity
2011-06-05 8:14 ` Avi Kivity
2011-06-05 13:45 ` Brad Campbell
2011-06-05 13:45 ` Brad Campbell
2011-06-05 13:58 ` Avi Kivity
2011-06-05 13:58 ` Avi Kivity
2011-06-06 20:22 ` Eric Dumazet
2011-06-06 20:22 ` Eric Dumazet
2011-06-06 20:22 ` Eric Dumazet
2011-06-07 13:27 ` Brad Campbell
2011-06-07 13:37 ` Eric Dumazet
2011-06-07 15:15 ` Brad Campbell
2011-08-20 13:16 ` Brad Campbell [this message]
2011-08-22 6:36 ` Avi Kivity
2011-08-22 6:45 ` Eric Dumazet
2011-08-22 11:45 ` Brad Campbell
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4E4FB3B7.4040501@fnarfbargle.com \
--to=lists2009@fnarfbargle.com \
--cc=avi@redhat.com \
--cc=bp@alien8.de \
--cc=cat@zip.com.au \
--cc=eric.dumazet@gmail.com \
--cc=kvm@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=netdev@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.