Re: [ovs-dev] Flow Control and Port Mirroring

Re: [ovs-dev] Flow Control and Port Mirroring

[Simon Horman]

[ CCed VHOST contacts ]

On Thu, Oct 28, 2010 at 01:22:02PM -0700, Jesse Gross wrote:
> On Thu, Oct 28, 2010 at 4:54 AM, Simon Horman <horms@verge.net.au> wrote:
> > My reasoning is that in the non-mirroring case the guest is
> > limited by the external interface through wich the packets
> > eventually flow - that is 1Gbit/s. But in the mirrored either
> > there is no flow control or the flow control is acting on the
> > rate of dummy0, which is essentailly infinate.
> >
> > Before investigating this any further I wanted to ask if
> > this behaviour is intentional.
> 
> It's not intentional but I can take a guess at what is happening.
> 
> When we send the packet to a mirror, the skb is cloned but only the
> original skb is charged to the sender.  If the original packet is
> delivered to localhost then it will be freed quickly and no longer
> accounted for, despite the fact that the "real" packet is still
> sitting in the transmit queue on the NIC.  The UDP stack will then
> send the next packet, limited only by the speed of the CPU.

That would explain what I have observed.

> Normally, this would be tracked by accounting for the memory charged
> to the socket.  However, I know that Xen tracks whether the actual
> pages of memory have been freed, which should avoid this problem since
> the memory won't be released util the last packet has been sent.  I
> don't know what KVM virtio does but I'm guessing that it similar to
> the former, since this problem is occurring.

I am also familiar of how Xen tracks pages but less sure of the
virtio side of things.

> While it would be easy to charge the socket for all clones, I also
> want to be careful about over accounting of the same data, leading to
> a very small effective socket buffer.

Agreed, we don't want to see over-charging.

Re: [ovs-dev] Flow Control and Port Mirroring

[Rusty Russell]

On Sat, 30 Oct 2010 01:29:33 pm Simon Horman wrote:
> [ CCed VHOST contacts ]
> 
> On Thu, Oct 28, 2010 at 01:22:02PM -0700, Jesse Gross wrote:
> > On Thu, Oct 28, 2010 at 4:54 AM, Simon Horman <horms@verge.net.au> wrote:
> > > My reasoning is that in the non-mirroring case the guest is
> > > limited by the external interface through wich the packets
> > > eventually flow - that is 1Gbit/s. But in the mirrored either
> > > there is no flow control or the flow control is acting on the
> > > rate of dummy0, which is essentailly infinate.
> > >
> > > Before investigating this any further I wanted to ask if
> > > this behaviour is intentional.
> > 
> > It's not intentional but I can take a guess at what is happening.
> > 
> > When we send the packet to a mirror, the skb is cloned but only the
> > original skb is charged to the sender.  If the original packet is
> > delivered to localhost then it will be freed quickly and no longer
> > accounted for, despite the fact that the "real" packet is still
> > sitting in the transmit queue on the NIC.  The UDP stack will then
> > send the next packet, limited only by the speed of the CPU.
> 
> That would explain what I have observed.

I can't find the thread (what is ovs-dev?), but I think the tap device
has this fundamental feature: you can blast as many packets as you want
through it.

If that's a bad thing, we have to look harder...

Cheers,
Rusty.

Re: [ovs-dev] Flow Control and Port Mirroring

[Simon Horman]

On Mon, Nov 08, 2010 at 01:41:23PM +1030, Rusty Russell wrote:
> On Sat, 30 Oct 2010 01:29:33 pm Simon Horman wrote:
> > [ CCed VHOST contacts ]
> > 
> > On Thu, Oct 28, 2010 at 01:22:02PM -0700, Jesse Gross wrote:
> > > On Thu, Oct 28, 2010 at 4:54 AM, Simon Horman <horms@verge.net.au> wrote:
> > > > My reasoning is that in the non-mirroring case the guest is
> > > > limited by the external interface through wich the packets
> > > > eventually flow - that is 1Gbit/s. But in the mirrored either
> > > > there is no flow control or the flow control is acting on the
> > > > rate of dummy0, which is essentailly infinate.
> > > >
> > > > Before investigating this any further I wanted to ask if
> > > > this behaviour is intentional.
> > > 
> > > It's not intentional but I can take a guess at what is happening.
> > > 
> > > When we send the packet to a mirror, the skb is cloned but only the
> > > original skb is charged to the sender.  If the original packet is
> > > delivered to localhost then it will be freed quickly and no longer
> > > accounted for, despite the fact that the "real" packet is still
> > > sitting in the transmit queue on the NIC.  The UDP stack will then
> > > send the next packet, limited only by the speed of the CPU.
> > 
> > That would explain what I have observed.
> 
> I can't find the thread (what is ovs-dev?),

Sorry, yes its on ovs-dev.
http://openvswitch.org/pipermail/dev_openvswitch.org/2010-October/003806.html

> but I think the tap device
> has this fundamental feature: you can blast as many packets as you want
> through it.
> 
> If that's a bad thing, we have to look harder...

There does seem to be flow control in the non-mirrored case.
So I suspect its occurring at the skb level but that breaks down when
a clone occurs. It would seem that fragment level flow control would
help this problem (which is basically what Xen's netback/netfront has),
but by this point I am speculating wildly.  I'll try and find out exactly
where the problem is occurring in order for us to have a more informed
discussion.