domU to domU networking issues in v3.7? (netserver/netperf failing to communicate)

domU to domU networking issues in v3.7? (netserver/netperf failing to communicate)

[Konrad Rzeszutek Wilk]

Hey Ian, Xen-devel mailingl list,

I think the issue of 70% traffic lost was actually introduced in v3.6 or
perhaps v3.5. Annie and Marcos (CC-ed here) are looking to see which of
the releases introduced this. The issue we are seeing is that a domU
to domU communication breaks - this is with netperf/netserver talking to
each other.

Anyhow, I think the 3.7 compound page exacerbated the problem and also
(at least on some of my test hardware) exposed existing issues with
drivers. The issue I have is that the 'skge' driver has a bug that has
been there for ages (I tested way back to 3.0 and still saw it) were it
can not work with SWIOTLB. It is probably missing an pci_dma_sync
somewhere. 

Anyhow the compound page got me to look at Xen-SWIOTLB and that looks
OK. Even with synthetic driver (the fake one I posted somewhere) it
dealt with compound pages properly (with debug or non-debug Xen
hypervisor).

So was wondering if you had looked at this in more details? Any
ideas? Or would it be more prudent to ask that once we know for sure
which Linux release introduced the communication failures between
guests?

Re: domU to domU networking issues in v3.7? (netserver/netperf failing to communicate)

[Ian Campbell]

On Sat, 2012-11-10 at 13:59 +0000, Konrad Rzeszutek Wilk wrote:
> Hey Ian, Xen-devel mailingl list,
> 
> I think the issue of 70% traffic lost was actually introduced in v3.6 or
> perhaps v3.5. Annie and Marcos (CC-ed here) are looking to see which of
> the releases introduced this. The issue we are seeing is that a domU
> to domU communication breaks - this is with netperf/netserver talking to
> each other.
> 
> Anyhow, I think the 3.7 compound page exacerbated the problem and also
> (at least on some of my test hardware) exposed existing issues with
> drivers. The issue I have is that the 'skge' driver has a bug that has
> been there for ages (I tested way back to 3.0 and still saw it) were it
> can not work with SWIOTLB. It is probably missing an pci_dma_sync
> somewhere. 
> 
> Anyhow the compound page got me to look at Xen-SWIOTLB and that looks
> OK. Even with synthetic driver (the fake one I posted somewhere) it
> dealt with compound pages properly (with debug or non-debug Xen
> hypervisor).

The debug build is probably most interesting since it deliberately
allocates a non 1-1 p-to-m mapping so as to catch exactly these sorts of
issues.

> So was wondering if you had looked at this in more details? Any
> ideas? Or would it be more prudent to ask that once we know for sure
> which Linux release introduced the communication failures between
> guests?

I've not looked at it any further I'm afraid.

If these changes (be they in 3.5 or later, or earlier) are exposing
driver bugs then I suspect the netdev chaps would want to know about it.

FWIW I see the issue with tg3.

Ian.

Re: domU to domU networking issues in v3.7? (netserver/netperf failing to communicate)

[Konrad Rzeszutek Wilk]

On Mon, Nov 12, 2012 at 09:54:44AM +0000, Ian Campbell wrote:
> On Sat, 2012-11-10 at 13:59 +0000, Konrad Rzeszutek Wilk wrote:
> > Hey Ian, Xen-devel mailingl list,
> > 
> > I think the issue of 70% traffic lost was actually introduced in v3.6 or
> > perhaps v3.5. Annie and Marcos (CC-ed here) are looking to see which of
> > the releases introduced this. The issue we are seeing is that a domU
> > to domU communication breaks - this is with netperf/netserver talking to
> > each other.
> > 
> > Anyhow, I think the 3.7 compound page exacerbated the problem and also
> > (at least on some of my test hardware) exposed existing issues with
> > drivers. The issue I have is that the 'skge' driver has a bug that has
> > been there for ages (I tested way back to 3.0 and still saw it) were it
> > can not work with SWIOTLB. It is probably missing an pci_dma_sync
> > somewhere. 
> > 
> > Anyhow the compound page got me to look at Xen-SWIOTLB and that looks
> > OK. Even with synthetic driver (the fake one I posted somewhere) it
> > dealt with compound pages properly (with debug or non-debug Xen
> > hypervisor).
> 
> The debug build is probably most interesting since it deliberately
> allocates a non 1-1 p-to-m mapping so as to catch exactly these sorts of
> issues.

Right. My test env runs with that. And so far it only has issues
with the skge one.
> 
> > So was wondering if you had looked at this in more details? Any
> > ideas? Or would it be more prudent to ask that once we know for sure
> > which Linux release introduced the communication failures between
> > guests?
> 
> I've not looked at it any further I'm afraid.
> 
> If these changes (be they in 3.5 or later, or earlier) are exposing
> driver bugs then I suspect the netdev chaps would want to know about it.

Right. Annie (CC-ed here) mentioned to me that v3.5 looks to work ok.
And is off checking v3.6. v3.7 is definitly a no go.
> 
> FWIW I see the issue with tg3.

I should double-check that but I didn't get any issues with the tg3 I had.

I was running the netperf/netserver from/to to see if I could reproduce the
issues (70% loss) you described but did not have much luck.

> 
> Ian.

Re: domU to domU networking issues in v3.7? (netserver/netperf failing to communicate)

[Ian Campbell]

> > FWIW I see the issue with tg3.
> 
> I should double-check that but I didn't get any issues with the tg3 I had.
> 
> I was running the netperf/netserver from/to to see if I could reproduce the
> issues (70% loss) you described but did not have much luck.

I was running from offbox to a domU, if that helps. It only happened in
one direction but I can't recall which.

Re: domU to domU networking issues in v3.7? (netserver/netperf failing to communicate)

[Sander Eikelenboom]

Monday, November 12, 2012, 3:28:35 PM, you wrote:

> On Mon, Nov 12, 2012 at 09:54:44AM +0000, Ian Campbell wrote:
>> On Sat, 2012-11-10 at 13:59 +0000, Konrad Rzeszutek Wilk wrote:
>> > Hey Ian, Xen-devel mailingl list,
>> > 
>> > I think the issue of 70% traffic lost was actually introduced in v3.6 or
>> > perhaps v3.5. Annie and Marcos (CC-ed here) are looking to see which of
>> > the releases introduced this. The issue we are seeing is that a domU
>> > to domU communication breaks - this is with netperf/netserver talking to
>> > each other.
>> > 
>> > Anyhow, I think the 3.7 compound page exacerbated the problem and also
>> > (at least on some of my test hardware) exposed existing issues with
>> > drivers. The issue I have is that the 'skge' driver has a bug that has
>> > been there for ages (I tested way back to 3.0 and still saw it) were it
>> > can not work with SWIOTLB. It is probably missing an pci_dma_sync
>> > somewhere. 
>> > 
>> > Anyhow the compound page got me to look at Xen-SWIOTLB and that looks
>> > OK. Even with synthetic driver (the fake one I posted somewhere) it
>> > dealt with compound pages properly (with debug or non-debug Xen
>> > hypervisor).
>> 
>> The debug build is probably most interesting since it deliberately
>> allocates a non 1-1 p-to-m mapping so as to catch exactly these sorts of
>> issues.

> Right. My test env runs with that. And so far it only has issues
> with the skge one.
>> 
>> > So was wondering if you had looked at this in more details? Any
>> > ideas? Or would it be more prudent to ask that once we know for sure
>> > which Linux release introduced the communication failures between
>> > guests?
>> 
>> I've not looked at it any further I'm afraid.
>> 
>> If these changes (be they in 3.5 or later, or earlier) are exposing
>> driver bugs then I suspect the netdev chaps would want to know about it.

> Right. Annie (CC-ed here) mentioned to me that v3.5 looks to work ok.
> And is off checking v3.6. v3.7 is definitly a no go.
>> 
>> FWIW I see the issue with tg3.

After the issues with netback where fixed, I'm seeing the issues with net_front reverting the single commit 5640f7685831e088fe6c2e1f863a6805962f8e81 (that was pointed out for netback) also makes these disappear.
I haven't seen issues with netfront and netback before this commit. FWIW: the hardware nics in this machine are two r8169 's.

--
Sander

> I should double-check that but I didn't get any issues with the tg3 I had.

> I was running the netperf/netserver from/to to see if I could reproduce the
> issues (70% loss) you described but did not have much luck.

>> 
>> Ian.

Re: domU to domU networking issues in v3.7? (netserver/netperf failing to communicate)

[Konrad Rzeszutek Wilk]

On Mon, Nov 12, 2012 at 03:50:24PM +0100, Sander Eikelenboom wrote:
> 
> Monday, November 12, 2012, 3:28:35 PM, you wrote:
> 
> > On Mon, Nov 12, 2012 at 09:54:44AM +0000, Ian Campbell wrote:
> >> On Sat, 2012-11-10 at 13:59 +0000, Konrad Rzeszutek Wilk wrote:
> >> > Hey Ian, Xen-devel mailingl list,
> >> > 
> >> > I think the issue of 70% traffic lost was actually introduced in v3.6 or
> >> > perhaps v3.5. Annie and Marcos (CC-ed here) are looking to see which of
> >> > the releases introduced this. The issue we are seeing is that a domU
> >> > to domU communication breaks - this is with netperf/netserver talking to
> >> > each other.
> >> > 
> >> > Anyhow, I think the 3.7 compound page exacerbated the problem and also
> >> > (at least on some of my test hardware) exposed existing issues with
> >> > drivers. The issue I have is that the 'skge' driver has a bug that has
> >> > been there for ages (I tested way back to 3.0 and still saw it) were it
> >> > can not work with SWIOTLB. It is probably missing an pci_dma_sync
> >> > somewhere. 
> >> > 
> >> > Anyhow the compound page got me to look at Xen-SWIOTLB and that looks
> >> > OK. Even with synthetic driver (the fake one I posted somewhere) it
> >> > dealt with compound pages properly (with debug or non-debug Xen
> >> > hypervisor).
> >> 
> >> The debug build is probably most interesting since it deliberately
> >> allocates a non 1-1 p-to-m mapping so as to catch exactly these sorts of
> >> issues.
> 
> > Right. My test env runs with that. And so far it only has issues
> > with the skge one.
> >> 
> >> > So was wondering if you had looked at this in more details? Any
> >> > ideas? Or would it be more prudent to ask that once we know for sure
> >> > which Linux release introduced the communication failures between
> >> > guests?
> >> 
> >> I've not looked at it any further I'm afraid.
> >> 
> >> If these changes (be they in 3.5 or later, or earlier) are exposing
> >> driver bugs then I suspect the netdev chaps would want to know about it.
> 
> > Right. Annie (CC-ed here) mentioned to me that v3.5 looks to work ok.
> > And is off checking v3.6. v3.7 is definitly a no go.
> >> 
> >> FWIW I see the issue with tg3.
> 
> After the issues with netback where fixed, I'm seeing the issues with net_front reverting the single commit 5640f7685831e088fe6c2e1f863a6805962f8e81 (that was pointed out for netback) also makes these disappear.

Were you ever able to trigger the BUG_ON in the patch that Ian posted?

> I haven't seen issues with netfront and netback before this commit. FWIW: the hardware nics in this machine are two r8169 's.
> 
> --
> Sander
> 
> > I should double-check that but I didn't get any issues with the tg3 I had.
> 
> > I was running the netperf/netserver from/to to see if I could reproduce the
> > issues (70% loss) you described but did not have much luck.
> 
> >> 
> >> Ian.
> 
> 

Re: domU to domU networking issues in v3.7? (netserver/netperf failing to communicate)

[Sander Eikelenboom]

Monday, November 12, 2012, 5:32:04 PM, you wrote:

> On Mon, Nov 12, 2012 at 03:50:24PM +0100, Sander Eikelenboom wrote:
>> 
>> Monday, November 12, 2012, 3:28:35 PM, you wrote:
>> 
>> > On Mon, Nov 12, 2012 at 09:54:44AM +0000, Ian Campbell wrote:
>> >> On Sat, 2012-11-10 at 13:59 +0000, Konrad Rzeszutek Wilk wrote:
>> >> > Hey Ian, Xen-devel mailingl list,
>> >> > 
>> >> > I think the issue of 70% traffic lost was actually introduced in v3.6 or
>> >> > perhaps v3.5. Annie and Marcos (CC-ed here) are looking to see which of
>> >> > the releases introduced this. The issue we are seeing is that a domU
>> >> > to domU communication breaks - this is with netperf/netserver talking to
>> >> > each other.
>> >> > 
>> >> > Anyhow, I think the 3.7 compound page exacerbated the problem and also
>> >> > (at least on some of my test hardware) exposed existing issues with
>> >> > drivers. The issue I have is that the 'skge' driver has a bug that has
>> >> > been there for ages (I tested way back to 3.0 and still saw it) were it
>> >> > can not work with SWIOTLB. It is probably missing an pci_dma_sync
>> >> > somewhere. 
>> >> > 
>> >> > Anyhow the compound page got me to look at Xen-SWIOTLB and that looks
>> >> > OK. Even with synthetic driver (the fake one I posted somewhere) it
>> >> > dealt with compound pages properly (with debug or non-debug Xen
>> >> > hypervisor).
>> >> 
>> >> The debug build is probably most interesting since it deliberately
>> >> allocates a non 1-1 p-to-m mapping so as to catch exactly these sorts of
>> >> issues.
>> 
>> > Right. My test env runs with that. And so far it only has issues
>> > with the skge one.
>> >> 
>> >> > So was wondering if you had looked at this in more details? Any
>> >> > ideas? Or would it be more prudent to ask that once we know for sure
>> >> > which Linux release introduced the communication failures between
>> >> > guests?
>> >> 
>> >> I've not looked at it any further I'm afraid.
>> >> 
>> >> If these changes (be they in 3.5 or later, or earlier) are exposing
>> >> driver bugs then I suspect the netdev chaps would want to know about it.
>> 
>> > Right. Annie (CC-ed here) mentioned to me that v3.5 looks to work ok.
>> > And is off checking v3.6. v3.7 is definitly a no go.
>> >> 
>> >> FWIW I see the issue with tg3.
>> 
>> After the issues with netback where fixed, I'm seeing the issues with net_front reverting the single commit 5640f7685831e088fe6c2e1f863a6805962f8e81 (that was pointed out for netback) also makes these disappear.

> Were you ever able to trigger the BUG_ON in the patch that Ian posted?

What exact patch (or any other patch that can help you ) ?
(so i can try again to be sure)


>> I haven't seen issues with netfront and netback before this commit. FWIW: the hardware nics in this machine are two r8169 's.
>> 
>> --
>> Sander
>> 
>> > I should double-check that but I didn't get any issues with the tg3 I had.
>> 
>> > I was running the netperf/netserver from/to to see if I could reproduce the
>> > issues (70% loss) you described but did not have much luck.
>> 
>> >> 
>> >> Ian.
>> 
>> 

Re: domU to domU networking issues in v3.7? (netserver/netperf failing to communicate)

[Konrad Rzeszutek Wilk]

On Mon, Nov 12, 2012 at 06:20:55PM +0100, Sander Eikelenboom wrote:
> 
> Monday, November 12, 2012, 5:32:04 PM, you wrote:
> 
> > On Mon, Nov 12, 2012 at 03:50:24PM +0100, Sander Eikelenboom wrote:
> >> 
> >> Monday, November 12, 2012, 3:28:35 PM, you wrote:
> >> 
> >> > On Mon, Nov 12, 2012 at 09:54:44AM +0000, Ian Campbell wrote:
> >> >> On Sat, 2012-11-10 at 13:59 +0000, Konrad Rzeszutek Wilk wrote:
> >> >> > Hey Ian, Xen-devel mailingl list,
> >> >> > 
> >> >> > I think the issue of 70% traffic lost was actually introduced in v3.6 or
> >> >> > perhaps v3.5. Annie and Marcos (CC-ed here) are looking to see which of
> >> >> > the releases introduced this. The issue we are seeing is that a domU
> >> >> > to domU communication breaks - this is with netperf/netserver talking to
> >> >> > each other.
> >> >> > 
> >> >> > Anyhow, I think the 3.7 compound page exacerbated the problem and also
> >> >> > (at least on some of my test hardware) exposed existing issues with
> >> >> > drivers. The issue I have is that the 'skge' driver has a bug that has
> >> >> > been there for ages (I tested way back to 3.0 and still saw it) were it
> >> >> > can not work with SWIOTLB. It is probably missing an pci_dma_sync
> >> >> > somewhere. 
> >> >> > 
> >> >> > Anyhow the compound page got me to look at Xen-SWIOTLB and that looks
> >> >> > OK. Even with synthetic driver (the fake one I posted somewhere) it
> >> >> > dealt with compound pages properly (with debug or non-debug Xen
> >> >> > hypervisor).
> >> >> 
> >> >> The debug build is probably most interesting since it deliberately
> >> >> allocates a non 1-1 p-to-m mapping so as to catch exactly these sorts of
> >> >> issues.
> >> 
> >> > Right. My test env runs with that. And so far it only has issues
> >> > with the skge one.
> >> >> 
> >> >> > So was wondering if you had looked at this in more details? Any
> >> >> > ideas? Or would it be more prudent to ask that once we know for sure
> >> >> > which Linux release introduced the communication failures between
> >> >> > guests?
> >> >> 
> >> >> I've not looked at it any further I'm afraid.
> >> >> 
> >> >> If these changes (be they in 3.5 or later, or earlier) are exposing
> >> >> driver bugs then I suspect the netdev chaps would want to know about it.
> >> 
> >> > Right. Annie (CC-ed here) mentioned to me that v3.5 looks to work ok.
> >> > And is off checking v3.6. v3.7 is definitly a no go.
> >> >> 
> >> >> FWIW I see the issue with tg3.
> >> 
> >> After the issues with netback where fixed, I'm seeing the issues with net_front reverting the single commit 5640f7685831e088fe6c2e1f863a6805962f8e81 (that was pointed out for netback) also makes these disappear.
> 
> > Were you ever able to trigger the BUG_ON in the patch that Ian posted?
> 
> What exact patch (or any other patch that can help you ) ?
> (so i can try again to be sure)

This one:
http://lists.xen.org/archives/html/xen-devel/2012-10/msg00893.html

Re: domU to domU networking issues in v3.7? (netserver/netperf failing to communicate)

[Sander Eikelenboom]

Wednesday, November 14, 2012, 7:24:53 AM, you wrote:

> On Mon, Nov 12, 2012 at 06:20:55PM +0100, Sander Eikelenboom wrote:
>> 
>> Monday, November 12, 2012, 5:32:04 PM, you wrote:
>> 
>> > On Mon, Nov 12, 2012 at 03:50:24PM +0100, Sander Eikelenboom wrote:
>> >> 
>> >> Monday, November 12, 2012, 3:28:35 PM, you wrote:
>> >> 
>> >> > On Mon, Nov 12, 2012 at 09:54:44AM +0000, Ian Campbell wrote:
>> >> >> On Sat, 2012-11-10 at 13:59 +0000, Konrad Rzeszutek Wilk wrote:
>> >> >> > Hey Ian, Xen-devel mailingl list,
>> >> >> > 
>> >> >> > I think the issue of 70% traffic lost was actually introduced in v3.6 or
>> >> >> > perhaps v3.5. Annie and Marcos (CC-ed here) are looking to see which of
>> >> >> > the releases introduced this. The issue we are seeing is that a domU
>> >> >> > to domU communication breaks - this is with netperf/netserver talking to
>> >> >> > each other.
>> >> >> > 
>> >> >> > Anyhow, I think the 3.7 compound page exacerbated the problem and also
>> >> >> > (at least on some of my test hardware) exposed existing issues with
>> >> >> > drivers. The issue I have is that the 'skge' driver has a bug that has
>> >> >> > been there for ages (I tested way back to 3.0 and still saw it) were it
>> >> >> > can not work with SWIOTLB. It is probably missing an pci_dma_sync
>> >> >> > somewhere. 
>> >> >> > 
>> >> >> > Anyhow the compound page got me to look at Xen-SWIOTLB and that looks
>> >> >> > OK. Even with synthetic driver (the fake one I posted somewhere) it
>> >> >> > dealt with compound pages properly (with debug or non-debug Xen
>> >> >> > hypervisor).
>> >> >> 
>> >> >> The debug build is probably most interesting since it deliberately
>> >> >> allocates a non 1-1 p-to-m mapping so as to catch exactly these sorts of
>> >> >> issues.
>> >> 
>> >> > Right. My test env runs with that. And so far it only has issues
>> >> > with the skge one.
>> >> >> 
>> >> >> > So was wondering if you had looked at this in more details? Any
>> >> >> > ideas? Or would it be more prudent to ask that once we know for sure
>> >> >> > which Linux release introduced the communication failures between
>> >> >> > guests?
>> >> >> 
>> >> >> I've not looked at it any further I'm afraid.
>> >> >> 
>> >> >> If these changes (be they in 3.5 or later, or earlier) are exposing
>> >> >> driver bugs then I suspect the netdev chaps would want to know about it.
>> >> 
>> >> > Right. Annie (CC-ed here) mentioned to me that v3.5 looks to work ok.
>> >> > And is off checking v3.6. v3.7 is definitly a no go.
>> >> >> 
>> >> >> FWIW I see the issue with tg3.
>> >> 
>> >> After the issues with netback where fixed, I'm seeing the issues with net_front reverting the single commit 5640f7685831e088fe6c2e1f863a6805962f8e81 (that was pointed out for netback) also makes these disappear.
>> 
>> > Were you ever able to trigger the BUG_ON in the patch that Ian posted?
>> 
>> What exact patch (or any other patch that can help you ) ?
>> (so i can try again to be sure)

> This one:
> http://lists.xen.org/archives/html/xen-devel/2012-10/msg00893.html


Sorry for the late response!
I have changed the bug_on in a warn_on and applied it to a kernel with and without commit 5640f7685831e088fe6c2e1f863a6805962f8e81 reverted.
- With commit 5640f7685831e088fe6c2e1f863a6805962f8e81 reverted I don't see any warnings (also after loading domU's with a few GB of network traffic)
- With a vanilla kernel and the patch, i can just ssh to a domU to see the warnings, after which SSH locks up. So it happens immediately, even without giving the domU any serious network load.


[   85.447191]  <IRQ>  [<ffffffff8106750a>] warn_slowpath_common+0x7a/0xb0
[   85.447201]  [<ffffffff81067555>] warn_slowpath_null+0x15/0x20
[   85.447207]  [<ffffffff814b2b6e>] xennet_start_xmit+0x7fe/0x860
[   85.447213]  [<ffffffff816a6169>] dev_hard_start_xmit+0x209/0x460
[   85.447220]  [<ffffffff816c1e56>] sch_direct_xmit+0xf6/0x290
[   85.447225]  [<ffffffff816a6566>] dev_queue_xmit+0x1a6/0x5a0
[   85.447231]  [<ffffffff816a63c0>] ? dev_hard_start_xmit+0x460/0x460
[   85.447238]  [<ffffffff816b0897>] neigh_resolve_output+0x127/0x250
[   85.447244]  [<ffffffff817405ad>] ip_finish_output+0x39d/0x530
[   85.447250]  [<ffffffff817402dd>] ? ip_finish_output+0xcd/0x530
[   85.447256]  [<ffffffff81740799>] ip_output+0x59/0xe0
[   85.447262]  [<ffffffff8173f2b8>] ip_local_out+0x28/0x90
[   85.447267]  [<ffffffff8173f86f>] ip_queue_xmit+0x17f/0x4a0
[   85.447273]  [<ffffffff8173f6f0>] ? ip_send_unicast_reply+0x340/0x340
[   85.447282]  [<ffffffff810a1cf7>] ? getnstimeofday+0x47/0xe0
[   85.447288]  [<ffffffff816962c9>] ? __skb_clone+0x29/0x120
[   85.447294]  [<ffffffff81755940>] tcp_transmit_skb+0x400/0x8d0
[   85.447303]  [<ffffffff8113ad55>] ? kmem_cache_free+0x125/0x1c0
[   85.450359]  [<ffffffff8175891a>] tcp_write_xmit+0x21a/0xa50
[   85.450359]  [<ffffffff81698c82>] ? __kfree_skb+0x42/0xa0
[   85.450359]  [<ffffffff817591bd>] __tcp_push_pending_frames+0x2d/0x90
[   85.450359]  [<ffffffff817545c4>] tcp_rcv_established+0x164/0x640
[   85.450359]  [<ffffffff8175cdbf>] ? tcp_v4_rcv+0x6cf/0xb10
[   85.450359]  [<ffffffff8175c3a5>] tcp_v4_do_rcv+0x135/0x480
[   85.450359]  [<ffffffff817d0dc2>] ? _raw_spin_lock_nested+0x42/0x50
[   85.450359]  [<ffffffff8175cdbf>] ? tcp_v4_rcv+0x6cf/0xb10
[   85.450359]  [<ffffffff8175d04d>] tcp_v4_rcv+0x95d/0xb10
[   85.450359]  [<ffffffff810b2158>] ? lock_acquire+0xd8/0x100
[   85.450359]  [<ffffffff81739895>] ? ip_local_deliver_finish+0x45/0x230
[   85.450359]  [<ffffffff8173996a>] ip_local_deliver_finish+0x11a/0x230
[   85.450359]  [<ffffffff81739895>] ? ip_local_deliver_finish+0x45/0x230
[   85.450359]  [<ffffffff81739ab8>] ip_local_deliver+0x38/0x80
[   85.450359]  [<ffffffff8173907a>] ip_rcv_finish+0x15a/0x630
[   85.450359]  [<ffffffff81739768>] ip_rcv+0x218/0x300
[   85.450359]  [<ffffffff816a1a9d>] __netif_receive_skb+0x65d/0x8d0
[   85.450359]  [<ffffffff816a1585>] ? __netif_receive_skb+0x145/0x8d0
[   85.450359]  [<ffffffff810ae3fd>] ? trace_hardirqs_on+0xd/0x10
[   85.450359]  [<ffffffff810faf83>] ? free_hot_cold_page+0x1b3/0x1e0
[   85.450359]  [<ffffffff816a4008>] netif_receive_skb+0x28/0xf0
[   85.450359]  [<ffffffff81699b13>] ? __pskb_pull_tail+0x253/0x340
[   85.450359]  [<ffffffff814b3795>] xennet_poll+0xad5/0xe10
[   85.450359]  [<ffffffff816a4db6>] net_rx_action+0x136/0x260
[   85.450359]  [<ffffffff8106f3a1>] ? __do_softirq+0x71/0x1a0
[   85.450359]  [<ffffffff8106f3f9>] __do_softirq+0xc9/0x1a0
[   85.450359]  [<ffffffff817d393c>] call_softirq+0x1c/0x30
[   85.450359]  [<ffffffff8100fd95>] do_softirq+0x85/0xf0
[   85.450359]  [<ffffffff8106f26e>] irq_exit+0x9e/0xd0
[   85.450359]  [<ffffffff8134659f>] xen_evtchn_do_upcall+0x2f/0x40
[   85.450359]  [<ffffffff817d399e>] xen_do_hypervisor_callback+0x1e/0x30
[   85.450359]  <EOI>  [<ffffffff8100122a>] ? xen_hypercall_xen_version+0xa/0x20
[   85.450359]  [<ffffffff8100122a>] ? xen_hypercall_xen_version+0xa/0x20
[   85.450359]  [<ffffffff8100880d>] ? xen_force_evtchn_callback+0xd/0x10
[   85.450359]  [<ffffffff810091b2>] ? check_events+0x12/0x20
[   85.450359]  [<ffffffff8100919f>] ? xen_restore_fl_direct_reloc+0x4/0x4
[   85.450359]  [<ffffffff810adfc0>] ? lockdep_trace_alloc+0xa0/0x130
[   85.450359]  [<ffffffff810fc9ad>] ? __alloc_pages_nodemask+0x8d/0x9a0
[   85.450359]  [<ffffffff810b2158>] ? lock_acquire+0xd8/0x100
[   85.450359]  [<ffffffff810b25e7>] ? lock_release+0x117/0x250
[   85.450359]  [<ffffffff812a332e>] ? check_blkcg_changed+0x8e/0x180
[   85.450359]  [<ffffffff812a32a0>] ? cfq_put_queue+0x220/0x220
[   85.450359]  [<ffffffff810b2158>] ? lock_acquire+0xd8/0x100
[   85.450359]  [<ffffffff81132f21>] ? alloc_pages_vma+0xc1/0x1d0
[   85.450359]  [<ffffffff8110752a>] ? shmem_alloc_page+0x5a/0x70
[   85.450359]  [<ffffffff810b25e7>] ? lock_release+0x117/0x250
[   85.450359]  [<ffffffff810f40fd>] ? find_get_page+0xad/0x100
[   85.450359]  [<ffffffff810f4050>] ? find_get_pages+0x190/0x190
[   85.450359]  [<ffffffff810b171b>] ? __lock_acquire+0x46b/0xdd0
[   85.450359]  [<ffffffff810f43d1>] ? find_lock_page+0x21/0x80
[   85.450359]  [<ffffffff81107c3f>] ? shmem_getpage_gfp+0x33f/0x720
[   85.450359]  [<ffffffff811080f6>] ? shmem_fault+0x46/0x70
[   85.450359]  [<ffffffff812a2d33>] ? cfq_add_cfqq_rr+0x33/0x100
[   85.450359]  [<ffffffff81269f79>] ? shm_fault+0x19/0x20
[   85.450359]  [<ffffffff81114de7>] ? __do_fault+0x67/0x4f0
[   85.450359]  [<ffffffff8111885d>] ? handle_pte_fault+0xfd/0x820
[   85.450359]  [<ffffffff8100509f>] ? pte_mfn_to_pfn+0x8f/0xf0
[   85.450359]  [<ffffffff81005159>] ? xen_pmd_val+0x9/0x10
[   85.450359]  [<ffffffff81119208>] ? handle_mm_fault+0x288/0x360
[   85.450359]  [<ffffffff8103eafc>] ? __do_page_fault+0x15c/0x510
[   85.450359]  [<ffffffff8128e2f3>] ? blk_finish_plug+0x13/0x50
[   85.450359]  [<ffffffff810ffc99>] ? __do_page_cache_readahead+0x229/0x2b0
[   85.450359]  [<ffffffff810ffb25>] ? __do_page_cache_readahead+0xb5/0x2b0
[   85.450359]  [<ffffffff810ac927>] ? trace_hardirqs_off_caller+0x77/0x150
[   85.450359]  [<ffffffff812b55ad>] ? trace_hardirqs_off_thunk+0x3a/0x3c
[   85.450359]  [<ffffffff8103eeb9>] ? do_page_fault+0x9/0x10
[   85.450359]  [<ffffffff817d20b8>] ? page_fault+0x28/0x30
[   85.450359]  [<ffffffff810f39a6>] ? file_read_actor+0x66/0x190
[   85.450359]  [<ffffffff81100230>] ? page_cache_async_readahead+0x80/0xa0
[   85.450359]  [<ffffffff810f4050>] ? find_get_pages+0x190/0x190
[   85.450359]  [<ffffffff810f6050>] ? generic_file_aio_read+0x2e0/0x6f0
[   85.450359]  [<ffffffff81144c82>] ? do_sync_read+0xa2/0xe0
[   85.450359]  [<ffffffff811459f5>] ? vfs_read+0xc5/0x190
[   85.450359]  [<ffffffff81145bca>] ? sys_read+0x5a/0xa0
[   85.450359]  [<ffffffff812b556e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[   85.450359]  [<ffffffff817d26a9>] ? system_call_fastpath+0x16/0x1b
[   85.450359] ---[ end trace ce03efc8ae5a5e97 ]---
[   85.650129] ------------[ cut here ]------------
[   85.650147] WARNING: at drivers/net/xen-netfront.c:465 xennet_start_xmit+0x7fe/0x860()
[   85.650154] Modules linked in:
[   85.650161] Pid: 0, comm: swapper/0 Tainted: G        W    3.7.0-rc5-20121116-vanilla-warn #1
[   85.650167] Call Trace:
[   85.650171]  <IRQ>  [<ffffffff8106750a>] warn_slowpath_common+0x7a/0xb0
[   85.650183]  [<ffffffff81067555>] warn_slowpath_null+0x15/0x20
[   85.650189]  [<ffffffff814b2b6e>] xennet_start_xmit+0x7fe/0x860
[   85.650196]  [<ffffffff816a6169>] dev_hard_start_xmit+0x209/0x460
[   85.650203]  [<ffffffff816c1e56>] sch_direct_xmit+0xf6/0x290
[   85.650208]  [<ffffffff816a6566>] dev_queue_xmit+0x1a6/0x5a0
[   85.650214]  [<ffffffff816a63c0>] ? dev_hard_start_xmit+0x460/0x460
[   85.650221]  [<ffffffff810b25e7>] ? lock_release+0x117/0x250
[   85.650227]  [<ffffffff81740436>] ip_finish_output+0x226/0x530
[   85.650234]  [<ffffffff817402dd>] ? ip_finish_output+0xcd/0x530
[   85.650240]  [<ffffffff81740799>] ip_output+0x59/0xe0
[   85.650245]  [<ffffffff8173f2b8>] ip_local_out+0x28/0x90
[   85.650251]  [<ffffffff8173f86f>] ip_queue_xmit+0x17f/0x4a0
[   85.650256]  [<ffffffff8173f6f0>] ? ip_send_unicast_reply+0x340/0x340
[   85.650262]  [<ffffffff810a1cf7>] ? getnstimeofday+0x47/0xe0
[   85.650268]  [<ffffffff816962c9>] ? __skb_clone+0x29/0x120
[   85.650274]  [<ffffffff81755940>] tcp_transmit_skb+0x400/0x8d0
[   85.650280]  [<ffffffff81758026>] tcp_retransmit_skb+0x1c6/0x5a0
[   85.650290]  [<ffffffff8175a300>] ? tcp_write_timer_handler+0x1a0/0x1a0
[   85.650296]  [<ffffffff81759e70>] tcp_retransmit_timer+0x370/0x660
[   85.650303]  [<ffffffff8175a29d>] tcp_write_timer_handler+0x13d/0x1a0
[   85.650309]  [<ffffffff8175a378>] tcp_write_timer+0x78/0x80
[   85.650315]  [<ffffffff81074f9c>] call_timer_fn+0x7c/0x100
[   85.650321]  [<ffffffff81074f20>] ? cascade+0xa0/0xa0
[   85.650326]  [<ffffffff8175a300>] ? tcp_write_timer_handler+0x1a0/0x1a0
[   85.650332]  [<ffffffff8175a300>] ? tcp_write_timer_handler+0x1a0/0x1a0
[   85.650338]  [<ffffffff81075237>] run_timer_softirq+0x217/0x250
[   85.650344]  [<ffffffff8106f3f9>] __do_softirq+0xc9/0x1a0
[   85.650350]  [<ffffffff817d393c>] call_softirq+0x1c/0x30
[   85.650356]  [<ffffffff8100fd95>] do_softirq+0x85/0xf0
[   85.650362]  [<ffffffff8106f26e>] irq_exit+0x9e/0xd0
[   85.650369]  [<ffffffff8134659f>] xen_evtchn_do_upcall+0x2f/0x40
[   85.650374]  [<ffffffff817d399e>] xen_do_hypervisor_callback+0x1e/0x30
[   85.650379]  <EOI>  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[   85.650389]  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[   85.650396]  [<ffffffff81008850>] ? xen_safe_halt+0x10/0x20
[   85.650402]  [<ffffffff810170f0>] ? default_idle+0x40/0x90
[   85.650408]  [<ffffffff810174a6>] ? cpu_idle+0x96/0xf0
[   85.650415]  [<ffffffff817a9c8c>] ? rest_init+0xbc/0xd0
[   85.650420]  [<ffffffff817a9bd0>] ? csum_partial_copy_generic+0x170/0x170
[   85.650428]  [<ffffffff81ce4be7>] ? start_kernel+0x390/0x39d
[   85.650433]  [<ffffffff81ce4677>] ? repair_env_string+0x5b/0x5b
[   85.650440]  [<ffffffff81ce4356>] ? x86_64_start_reservations+0x131/0x136
[   85.650446]  [<ffffffff81ce7915>] ? xen_start_kernel+0x54e/0x550
[   85.650451] ---[ end trace ce03efc8ae5a5e98 ]---
[   85.806018] ------------[ cut here ]------------
[   85.806044] WARNING: at drivers/net/xen-netfront.c:465 xennet_start_xmit+0x7fe/0x860()
[   85.806051] Modules linked in:
[   85.806058] Pid: 2019, comm: postgres Tainted: G        W    3.7.0-rc5-20121116-vanilla-warn #1
[   85.806065] Call Trace:
[   85.806073]  [<ffffffff8106750a>] warn_slowpath_common+0x7a/0xb0
[   85.806080]  [<ffffffff81067555>] warn_slowpath_null+0x15/0x20
[   85.806086]  [<ffffffff814b2b6e>] xennet_start_xmit+0x7fe/0x860
[   85.806103]  [<ffffffff816a6169>] dev_hard_start_xmit+0x209/0x460
[   85.806109]  [<ffffffff816c1e56>] sch_direct_xmit+0xf6/0x290
[   85.806114]  [<ffffffff816a6566>] dev_queue_xmit+0x1a6/0x5a0
[   85.806119]  [<ffffffff816a63c0>] ? dev_hard_start_xmit+0x460/0x460
[   85.806125]  [<ffffffff810abab5>] ? trace_softirqs_off+0x85/0x1b0
[   85.806132]  [<ffffffff81740436>] ip_finish_output+0x226/0x530
[   85.806138]  [<ffffffff817402dd>] ? ip_finish_output+0xcd/0x530
[   85.806143]  [<ffffffff81740799>] ip_output+0x59/0xe0
[   85.806148]  [<ffffffff8173f2b8>] ip_local_out+0x28/0x90
[   85.806153]  [<ffffffff8173f86f>] ip_queue_xmit+0x17f/0x4a0
[   85.806157]  [<ffffffff8173f6f0>] ? ip_send_unicast_reply+0x340/0x340
[   85.806163]  [<ffffffff810a1cf7>] ? getnstimeofday+0x47/0xe0
[   85.806168]  [<ffffffff816962c9>] ? __skb_clone+0x29/0x120
[   85.806174]  [<ffffffff81755940>] tcp_transmit_skb+0x400/0x8d0
[   85.806179]  [<ffffffff8175891a>] tcp_write_xmit+0x21a/0xa50
[   85.806184]  [<ffffffff8175917b>] tcp_push_one+0x2b/0x40
[   85.806190]  [<ffffffff81749cdc>] tcp_sendmsg+0x8dc/0xe20
[   85.806195]  [<ffffffff8176fe09>] inet_sendmsg+0xa9/0x100
[   85.806200]  [<ffffffff8176fd60>] ? inet_autobind+0x70/0x70
[   85.806206]  [<ffffffff8168dd12>] sock_sendmsg+0x82/0xb0
[   85.806211]  [<ffffffff8168ddb5>] ? sockfd_lookup_light+0x25/0x80
[   85.806216]  [<ffffffff8168e5e4>] sys_sendto+0x104/0x140
[   85.806223]  [<ffffffff817d26a9>] system_call_fastpath+0x16/0x1b
[   85.806227] ---[ end trace ce03efc8ae5a5e99 ]---
[   85.806855] ------------[ cut here ]------------
[   85.806871] WARNING: at drivers/net/xen-netfront.c:465 xennet_start_xmit+0x7fe/0x860()
[   85.806879] Modules linked in:
[   85.806884] Pid: 2019, comm: postgres Tainted: G        W    3.7.0-rc5-20121116-vanilla-warn #1
[   85.806891] Call Trace:
[   85.806898]  [<ffffffff8106750a>] warn_slowpath_common+0x7a/0xb0
[   85.806905]  [<ffffffff81067555>] warn_slowpath_null+0x15/0x20
[   85.806911]  [<ffffffff814b2b6e>] xennet_start_xmit+0x7fe/0x860
[   85.806917]  [<ffffffff816a6169>] dev_hard_start_xmit+0x209/0x460
[   85.806923]  [<ffffffff816c1e56>] sch_direct_xmit+0xf6/0x290
[   85.806929]  [<ffffffff816a6566>] dev_queue_xmit+0x1a6/0x5a0
[   85.806935]  [<ffffffff816a63c0>] ? dev_hard_start_xmit+0x460/0x460
[   85.806941]  [<ffffffff810b25e7>] ? lock_release+0x117/0x250
[   85.806947]  [<ffffffff81740436>] ip_finish_output+0x226/0x530
[   85.806953]  [<ffffffff817402dd>] ? ip_finish_output+0xcd/0x530
[   85.806959]  [<ffffffff81740799>] ip_output+0x59/0xe0
[   85.806964]  [<ffffffff8173f2b8>] ip_local_out+0x28/0x90
[   85.806970]  [<ffffffff8173f86f>] ip_queue_xmit+0x17f/0x4a0
[   85.806975]  [<ffffffff8173f6f0>] ? ip_send_unicast_reply+0x340/0x340
[   85.806982]  [<ffffffff810a1cf7>] ? getnstimeofday+0x47/0xe0
[   85.806987]  [<ffffffff816962c9>] ? __skb_clone+0x29/0x120
[   85.806993]  [<ffffffff81755940>] tcp_transmit_skb+0x400/0x8d0
[   85.806999]  [<ffffffff81758026>] tcp_retransmit_skb+0x1c6/0x5a0
[   85.807005]  [<ffffffff81094806>] ? finish_task_switch+0x46/0xe0
[   85.807011]  [<ffffffff8175859e>] tcp_xmit_retransmit_queue+0x19e/0x300
[   85.807018]  [<ffffffff8175070b>] tcp_fastretrans_alert+0x76b/0xcb0
[   85.807024]  [<ffffffff817515fc>] tcp_ack+0x9ac/0x1150
[   85.807030]  [<ffffffff81752663>] ? tcp_validate_incoming+0x2d3/0x350
[   85.807036]  [<ffffffff81754778>] tcp_rcv_established+0x318/0x640
[   85.807042]  [<ffffffff8106f776>] ? local_bh_enable+0xc6/0x160
[   85.807048]  [<ffffffff8175c3a5>] tcp_v4_do_rcv+0x135/0x480
[   85.807054]  [<ffffffff8106f08f>] ? __local_bh_disable+0x8f/0xe0
[   85.807061]  [<ffffffff817483db>] tcp_prequeue_process+0x6b/0xa0
[   85.807067]  [<ffffffff81748e40>] tcp_recvmsg+0x8f0/0xb70
[   85.807072]  [<ffffffff810b25e7>] ? lock_release+0x117/0x250
[   85.807078]  [<ffffffff8176ec2a>] inet_recvmsg+0xba/0x100
[   85.807084]  [<ffffffff8176fd60>] ? inet_autobind+0x70/0x70
[   85.807090]  [<ffffffff8168dbf9>] sock_recvmsg+0x89/0xc0
[   85.807096]  [<ffffffff81116b0f>] ? tlb_flush_mmu+0x5f/0xa0
[   85.807102]  [<ffffffff810b171b>] ? __lock_acquire+0x46b/0xdd0
[   85.807108]  [<ffffffff8168ddb5>] ? sockfd_lookup_light+0x25/0x80
[   85.807114]  [<ffffffff8168e45f>] sys_recvfrom+0xbf/0x120
[   85.807120]  [<ffffffff811144db>] ? might_fault+0x3b/0x90
[   85.807126]  [<ffffffff810b25e7>] ? lock_release+0x117/0x250
[   85.807133]  [<ffffffff817d26a9>] system_call_fastpath+0x16/0x1b
[   85.807138] ---[ end trace ce03efc8ae5a5e9a ]---
[   85.807159] ------------[ cut here ]------------
[   85.807165] WARNING: at drivers/net/xen-netfront.c:465 xennet_start_xmit+0x7fe/0x860()
[   85.807171] Modules linked in:
[   85.807176] Pid: 2019, comm: postgres Tainted: G        W    3.7.0-rc5-20121116-vanilla-warn #1
[   85.807182] Call Trace:
[   85.807187]  [<ffffffff8106750a>] warn_slowpath_common+0x7a/0xb0
[   85.807193]  [<ffffffff81067555>] warn_slowpath_null+0x15/0x20
[   85.807199]  [<ffffffff814b2b6e>] xennet_start_xmit+0x7fe/0x860
[   85.807205]  [<ffffffff816a6169>] dev_hard_start_xmit+0x209/0x460
[   85.807211]  [<ffffffff816c1e56>] sch_direct_xmit+0xf6/0x290
[   85.807216]  [<ffffffff816a6566>] dev_queue_xmit+0x1a6/0x5a0
[   85.807222]  [<ffffffff816a63c0>] ? dev_hard_start_xmit+0x460/0x460
[   85.807228]  [<ffffffff810b25e7>] ? lock_release+0x117/0x250
[   85.807233]  [<ffffffff81740436>] ip_finish_output+0x226/0x530
[   85.807239]  [<ffffffff817402dd>] ? ip_finish_output+0xcd/0x530
[   85.807245]  [<ffffffff81740799>] ip_output+0x59/0xe0
[   85.807250]  [<ffffffff8173f2b8>] ip_local_out+0x28/0x90
[   85.807256]  [<ffffffff8173f86f>] ip_queue_xmit+0x17f/0x4a0
[   85.807261]  [<ffffffff8173f6f0>] ? ip_send_unicast_reply+0x340/0x340
[   85.807267]  [<ffffffff810a1cf7>] ? getnstimeofday+0x47/0xe0
[   85.807272]  [<ffffffff816962c9>] ? __skb_clone+0x29/0x120
[   85.807278]  [<ffffffff81755940>] tcp_transmit_skb+0x400/0x8d0
[   85.807283]  [<ffffffff81758026>] tcp_retransmit_skb+0x1c6/0x5a0
[   85.810124]  [<ffffffff81094806>] ? finish_task_switch+0x46/0xe0
[   85.810124]  [<ffffffff8175859e>] tcp_xmit_retransmit_queue+0x19e/0x300
[   85.810124]  [<ffffffff8175070b>] tcp_fastretrans_alert+0x76b/0xcb0
[   85.810124]  [<ffffffff817515fc>] tcp_ack+0x9ac/0x1150
[   85.810124]  [<ffffffff81752663>] ? tcp_validate_incoming+0x2d3/0x350
[   85.810124]  [<ffffffff81754778>] tcp_rcv_established+0x318/0x640
[   85.810124]  [<ffffffff8106f776>] ? local_bh_enable+0xc6/0x160
[   85.810124]  [<ffffffff8175c3a5>] tcp_v4_do_rcv+0x135/0x480
[   85.810124]  [<ffffffff8106f08f>] ? __local_bh_disable+0x8f/0xe0
[   85.810124]  [<ffffffff817483db>] tcp_prequeue_process+0x6b/0xa0
[   85.810124]  [<ffffffff81748e40>] tcp_recvmsg+0x8f0/0xb70
[   85.810124]  [<ffffffff810b25e7>] ? lock_release+0x117/0x250
[   85.810124]  [<ffffffff8176ec2a>] inet_recvmsg+0xba/0x100
[   85.810124]  [<ffffffff8176fd60>] ? inet_autobind+0x70/0x70
[   85.810124]  [<ffffffff8168dbf9>] sock_recvmsg+0x89/0xc0
[   85.810124]  [<ffffffff81116b0f>] ? tlb_flush_mmu+0x5f/0xa0
[   85.8