From mboxrd@z Thu Jan  1 00:00:00 1970
From: Nivedita Singhvi <niv@us.ibm.com>
Subject: Re: bug: slab corruption (net backend?)
Date: Thu, 07 Jul 2005 09:06:12 -0700
Message-ID: <42CD52F4.2010407@us.ibm.com>
References: <20050707094132.GA29339@bytesex>	<E1DqTLV-00068D-00@mta1.cl.cam.ac.uk>
	<20050707113514.GA30556@bytesex>	<f9095f4e739734ec5b54796ebe6f31ae@cl.cam.ac.uk>	<20050707122242.GA31229@bytesex>
	<87br5ehi15.fsf@bytesex.org>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit
Return-path: <xen-devel-bounces@lists.xensource.com>
In-Reply-To: <87br5ehi15.fsf@bytesex.org>
List-Unsubscribe: <http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-devel>,
	<mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
List-Post: <mailto:xen-devel@lists.xensource.com>
List-Help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-Subscribe: <http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-devel>,
	<mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
Sender: xen-devel-bounces@lists.xensource.com
Errors-To: xen-devel-bounces@lists.xensource.com
To: Gerd Knorr <kraxel@suse.de>
Cc: xen-devel@lists.xensource.com, Steven Hand <Steven.Hand@cl.cam.ac.uk>
List-Id: xen-devel@lists.xenproject.org

Gerd Knorr wrote:

>   Hi,
> 
> Maybe related:  I also see tcp connection stalls.  Just booting domain0
> is enougth for that, whereas I see the slab corruption stuff only
> after trying to boot some domU.
> 
> As the tcpdump below shows the xen machine sends the same packet over
> and over again.  I'd guess some kind of memory corruption which kills
> the packet checksum and makes eskarina drop the packet.  Maybe just
> use-after-free as the slabdebug stuff will fill released memory blocks
> with some pattern.

Gerd,

Can you disable tcp checksum offload (ethtool -K tx off) and see
if you can reproduce this problem?

thanks,
Nivedita


> 14:56:41.445799 IP eskarina.40106 > master-xen.ssh: S 4050091082:4050091082(0) win 5840 <mss 1460,sackOK,timestamp 2080736744 0,nop,wscale 2>
> 14:56:41.446106 IP master-xen.ssh > eskarina.40106: S 892717242:892717242(0) ack 4050091083 win 5792 <mss 1460,sackOK,timestamp 5230 2080736744,nop,wscale 2>
> 14:56:41.446129 IP eskarina.40106 > master-xen.ssh: . ack 1 win 1460 <nop,nop,timestamp 2080736744 5230>
> 14:56:41.461316 IP master-xen.ssh > eskarina.40106: P 1:24(23) ack 1 win 1448 <nop,nop,timestamp 5232 2080736744>
> 14:56:41.461720 IP eskarina.40106 > master-xen.ssh: . ack 24 win 1460 <nop,nop,timestamp 2080736760 5232>
> 14:56:41.462250 IP eskarina.40106 > master-xen.ssh: P 1:23(22) ack 24 win 1460 <nop,nop,timestamp 2080736760 5232>
> 14:56:41.462641 IP master-xen.ssh > eskarina.40106: . ack 23 win 1448 <nop,nop,timestamp 5232 2080736760>
> 14:56:41.463196 IP eskarina.40106 > master-xen.ssh: P 23:663(640) ack 24 win 1460 <nop,nop,timestamp 2080736761 5232>
> 14:56:41.463988 IP master-xen.ssh > eskarina.40106: . ack 663 win 1768 <nop,nop,timestamp 5232 2080736761>
> 14:56:41.464835 IP master-xen.ssh > eskarina.40106: P 24:664(640) ack 663 win 1768 <nop,nop,timestamp 5232 2080736761>
> 14:56:41.465183 IP eskarina.40106 > master-xen.ssh: P 663:687(24) ack 664 win 1780 <nop,nop,timestamp 2080736763 5232>
> 14:56:41.469605 IP master-xen.ssh > eskarina.40106: P 664:816(152) ack 687 win 1768 <nop,nop,timestamp 5233 2080736763>
> 14:56:41.485010 IP eskarina.40106 > master-xen.ssh: P 687:831(144) ack 816 win 2100 <nop,nop,timestamp 2080736783 5233>
> 14:56:41.493626 IP master-xen.ssh > eskarina.40106: P 816:1280(464) ack 831 win 1768 <nop,nop,timestamp 5235 2080736783>
> 14:56:41.495546 IP eskarina.40106 > master-xen.ssh: P 831:847(16) ack 1280 win 2420 <nop,nop,timestamp 2080736793 5235>
> 14:56:41.529119 IP master-xen.ssh > eskarina.40106: . ack 847 win 1768 <nop,nop,timestamp 5239 2080736793>
> 14:56:41.529143 IP eskarina.40106 > master-xen.ssh: P 847:895(48) ack 1280 win 2420 <nop,nop,timestamp 2080736827 5239>
> 14:56:41.529488 IP master-xen.ssh > eskarina.40106: . ack 895 win 1768 <nop,nop,timestamp 5239 2080736827>
> 14:56:41.529642 IP master-xen.ssh > eskarina.40106: P 1280:1328(48) ack 895 win 1768 <nop,nop,timestamp 5239 2080736827>
> 14:56:41.530430 IP eskarina.40106 > master-xen.ssh: P 895:959(64) ack 1328 win 2420 <nop,nop,timestamp 2080736828 5239>
> 14:56:41.534746 IP master-xen.ssh > eskarina.40106: P 1328:1392(64) ack 959 win 1768 <nop,nop,timestamp 5239 2080736828>
> 14:56:41.534918 IP eskarina.40106 > master-xen.ssh: P 959:1199(240) ack 1392 win 2420 <nop,nop,timestamp 2080736833 5239>
> 14:56:41.536297 IP master-xen.ssh > eskarina.40106: P 1392:1584(192) ack 1199 win 2088 <nop,nop,timestamp 5239 2080736833>
> 14:56:41.538960 IP eskarina.40106 > master-xen.ssh: P 1199:1583(384) ack 1584 win 2420 <nop,nop,timestamp 2080736837 5239>
> 14:56:41.541838 IP master-xen.ssh > eskarina.40106: P 1584:1616(32) ack 1583 win 2408 <nop,nop,timestamp 5240 2080736837>
> 14:56:41.542436 IP eskarina.40106 > master-xen.ssh: P 1583:1647(64) ack 1616 win 2420 <nop,nop,timestamp 2080736840 5240>
> 14:56:41.579115 IP master-xen.ssh > eskarina.40106: . ack 1647 win 2408 <nop,nop,timestamp 5244 2080736840>
> 14:56:41.580513 IP master-xen.ssh > eskarina.40106: P 1616:1664(48) ack 1647 win 2408 <nop,nop,timestamp 5244 2080736840>
> 14:56:41.581070 IP eskarina.40106 > master-xen.ssh: P 1647:2287(640) ack 1664 win 2420 <nop,nop,timestamp 2080736879 5244>
> 14:56:41.581859 IP master-xen.ssh > eskarina.40106: . ack 2287 win 2728 <nop,nop,timestamp 5244 2080736879>
> 14:56:41.613880 IP master-xen.ssh > eskarina.40106: P 1664:1712(48) ack 2287 win 2728 <nop,nop,timestamp 5247 2080736879>
> 14:56:41.614052 IP master-xen.ssh > eskarina.40106: P 1712:1840(128) ack 2287 win 2728 <nop,nop,timestamp 5247 2080736879>
> 14:56:41.653594 IP eskarina.40106 > master-xen.ssh: . ack 1840 win 2740 <nop,nop,timestamp 2080736952 5247>
> 14:56:41.662633 IP master-xen.ssh > eskarina.40106: P 1840:1904(64) ack 2287 win 2728 <nop,nop,timestamp 5252 2080736952>
> 14:56:41.662697 IP eskarina.40106 > master-xen.ssh: . ack 1904 win 2740 <nop,nop,timestamp 2080736961 5252>
> 14:56:41.668714 IP master-xen.ssh > eskarina.40106: P 1904:1984(80) ack 2287 win 2728 <nop,nop,timestamp 5252 2080736961>
> 14:56:41.668798 IP eskarina.40106 > master-xen.ssh: . ack 1984 win 2740 <nop,nop,timestamp 2080736967 5252>
> 14:56:41.668836 IP master-xen.ssh > eskarina.40106: P 1984:2048(64) ack 2287 win 2728 <nop,nop,timestamp 5252 2080736961>
> 14:56:41.668874 IP eskarina.40106 > master-xen.ssh: . ack 2048 win 2740 <nop,nop,timestamp 2080736967 5252>
> 14:56:41.669064 IP master-xen.ssh > eskarina.40106: P 2048:2096(48) ack 2287 win 2728 <nop,nop,timestamp 5252 2080736961>
> 14:56:41.669254 IP eskarina.40106 > master-xen.ssh: . ack 2096 win 2740 <nop,nop,timestamp 2080736967 5252>
> 14:56:41.670240 IP master-xen.ssh > eskarina.40106: P 2096:2160(64) ack 2287 win 2728 <nop,nop,timestamp 5253 2080736967>
> 14:56:41.671049 IP eskarina.40106 > master-xen.ssh: . ack 2160 win 2740 <nop,nop,timestamp 2080736969 5253>
> 14:56:43.375106 IP eskarina.40106 > master-xen.ssh: P 2287:2335(48) ack 2160 win 2740 <nop,nop,timestamp 2080738673 5253>
> 14:56:43.376098 IP master-xen.ssh > eskarina.40106: P 2160:2208(48) ack 2335 win 2728 <nop,nop,timestamp 5423 2080738673>
> 14:56:43.376119 IP eskarina.40106 > master-xen.ssh: . ack 2208 win 2740 <nop,nop,timestamp 2080738674 5423>
> 14:56:43.518162 IP eskarina.40106 > master-xen.ssh: P 2335:2383(48) ack 2208 win 2740 <nop,nop,timestamp 2080738816 5423>
> 14:56:43.518924 IP master-xen.ssh > eskarina.40106: P 2208:2256(48) ack 2383 win 2728 <nop,nop,timestamp 5437 2080738816>
> 14:56:43.518942 IP eskarina.40106 > master-xen.ssh: . ack 2256 win 2740 <nop,nop,timestamp 2080738817 5437>
> 14:56:43.599847 IP eskarina.40106 > master-xen.ssh: P 2383:2431(48) ack 2256 win 2740 <nop,nop,timestamp 2080738898 5437>
> 14:56:43.600591 IP master-xen.ssh > eskarina.40106: P 2256:2304(48) ack 2431 win 2728 <nop,nop,timestamp 5446 2080738898>
> 14:56:43.600607 IP eskarina.40106 > master-xen.ssh: . ack 2304 win 2740 <nop,nop,timestamp 2080738899 5446>
> 14:56:43.773975 IP eskarina.40106 > master-xen.ssh: P 2431:2479(48) ack 2304 win 2740 <nop,nop,timestamp 2080739072 5446>
> 14:56:43.774815 IP master-xen.ssh > eskarina.40106: P 2304:2352(48) ack 2479 win 2728 <nop,nop,timestamp 5463 2080739072>
> 14:56:43.774834 IP eskarina.40106 > master-xen.ssh: . ack 2352 win 2740 <nop,nop,timestamp 2080739073 5463>
> 14:56:43.843676 IP eskarina.40106 > master-xen.ssh: P 2479:2527(48) ack 2352 win 2740 <nop,nop,timestamp 2080739142 5463>
> 14:56:43.844417 IP master-xen.ssh > eskarina.40106: P 2352:2400(48) ack 2527 win 2728 <nop,nop,timestamp 5470 2080739142>
> 14:56:43.844601 IP eskarina.40106 > master-xen.ssh: . ack 2400 win 2740 <nop,nop,timestamp 2080739143 5470>
> 14:56:44.200677 IP eskarina.40106 > master-xen.ssh: P 2527:2575(48) ack 2400 win 2740 <nop,nop,timestamp 2080739499 5470>
> 14:56:44.201595 IP master-xen.ssh > eskarina.40106: P 2400:2448(48) ack 2575 win 2728 <nop,nop,timestamp 5506 2080739499>
> 14:56:44.201616 IP eskarina.40106 > master-xen.ssh: . ack 2448 win 2740 <nop,nop,timestamp 2080739500 5506>
> 14:56:44.207227 IP master-xen.ssh > eskarina.40106: . 2448:3896(1448) ack 2575 win 2728 <nop,nop,timestamp 5506 2080739500>
> 14:56:44.207255 IP eskarina.40106 > master-xen.ssh: . ack 3896 win 3464 <nop,nop,timestamp 2080739506 5506>
> 14:56:44.208455 IP master-xen.ssh > eskarina.40106: . 3896:5344(1448) ack 2575 win 2728 <nop,nop,timestamp 5506 2080739500>
> 14:56:44.208475 IP eskarina.40106 > master-xen.ssh: . ack 5344 win 4188 <nop,nop,timestamp 2080739507 5506>
> 14:56:44.209923 IP master-xen.ssh > eskarina.40106: . 5344:6792(1448) ack 2575 win 2728 <nop,nop,timestamp 5506 2080739506>
> 14:56:44.211154 IP master-xen.ssh > eskarina.40106: . 6792:8240(1448) ack 2575 win 2728 <nop,nop,timestamp 5506 2080739506>
> 14:56:44.211174 IP eskarina.40106 > master-xen.ssh: . ack 5344 win 4188 <nop,nop,timestamp 2080739510 5506,nop,nop,sack sack 1 {6792:8240} >
> 14:56:44.212383 IP master-xen.ssh > eskarina.40106: . 8240:9688(1448) ack 2575 win 2728 <nop,nop,timestamp 5506 2080739507>
> 14:56:44.213790 IP master-xen.ssh > eskarina.40106: . 9688:11136(1448) ack 2575 win 2728 <nop,nop,timestamp 5506 2080739507>
> 14:56:44.213798 IP eskarina.40106 > master-xen.ssh: . ack 5344 win 4188 <nop,nop,timestamp 2080739512 5506,nop,nop,sack sack 2 {9688:11136}{6792:8240} >
> 14:56:44.215022 IP master-xen.ssh > eskarina.40106: . 11136:12584(1448) ack 2575 win 2728 <nop,nop,timestamp 5507 2080739510>
> 14:56:44.215034 IP eskarina.40106 > master-xen.ssh: . ack 5344 win 4188 <nop,nop,timestamp 2080739513 5506,nop,nop,sack sack 2 {9688:12584}{6792:8240} >
> 14:56:44.216438 IP master-xen.ssh > eskarina.40106: . 5344:6792(1448) ack 2575 win 2728 <nop,nop,timestamp 5507 2080739512>
> 14:56:44.420361 IP master-xen.ssh > eskarina.40106: . 5344:6792(1448) ack 2575 win 2728 <nop,nop,timestamp 5528 2080739513>
> 14:56:44.840346 IP master-xen.ssh > eskarina.40106: . 5344:6792(1448) ack 2575 win 2728 <nop,nop,timestamp 5570 2080739513>
> 14:56:45.680357 IP master-xen.ssh > eskarina.40106: . 5344:6792(1448) ack 2575 win 2728 <nop,nop,timestamp 5654 2080739513>
> 14:56:47.360382 IP master-xen.ssh > eskarina.40106: . 5344:6792(1448) ack 2575 win 2728 <nop,nop,timestamp 5822 2080739513>
> 14:56:50.720720 IP master-xen.ssh > eskarina.40106: . 5344:6792(1448) ack 2575 win 2728 <nop,nop,timestamp 6158 2080739513>
> 14:56:57.440528 IP master-xen.ssh > eskarina.40106: . 5344:6792(1448) ack 2575 win 2728 <nop,nop,timestamp 6830 2080739513>
> 14:57:10.880688 IP master-xen.ssh > eskarina.40106: . 5344:6792(1448) ack 2575 win 2728 <nop,nop,timestamp 8174 2080739513>
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 


-- 
---
Nivedita Singhvi (nivedita@us.ibm.com Lotus)
Niv Singhvi (niv@us.ibm.com IMAP)
(503) 578-4580
T/L 775-4580