Trouble with TCP between domUs

Trouble with TCP between domUs

[Jacob Gorm Hansen]

hi,

I have added a network driver and the UIP tiny TCP/IP stack to mini-os,
and I am trying to network between mini-os and a domU running XenLinux.

>From dom0 I can ping and TCP connect to the domain just fine, but from
the domU the TCP connection hangs after the first few packets. UIP
complains about missing TCP checksums on the incoming packets, but I
think I am having trouble with MTUs > PAGE_SIZE as well. I am assuming
that what bites me is the various domU->domU optimizations being applied
in XenLinux netfront.

How can I turn off all the fancy optimizations in XenLinux netfront in
one go, in order to make progress without all these features in my
driver, or alternatively, is there a required minimum spec that I must
adhere to? As I said, things work just fine when networking with dom0.

Thanks,
Jacob

Re: Trouble with TCP between domUs

[Ian Campbell]

On Thu, 2006-12-07 at 16:38 +0100, Jacob Gorm Hansen wrote:
> >From dom0 I can ping and TCP connect to the domain just fine, but from
> the domU the TCP connection hangs after the first few packets. UIP
> complains about missing TCP checksums on the incoming packets,

You can get round that one by writing a feature-no-csum-offload=1 node
in xenstore. netfront.c in the sparse tree does this conditionally in
order to support older Linux kernels that also don't cope well with
missing checksums. With that set dom0 should do checksumming for you.

>  but I
> think I am having trouble with MTUs > PAGE_SIZE as well.

I'd have thought that not setting "feature-sg" in xenstore would do that
but I'm not 100% sure.

Perhaps you need to set MTU on vifX.Y in dom0 too?

Ian.

Re: Trouble with TCP between domUs

[David Edmondson]

* Ian.Campbell@XenSource.com [2006-12-07 15:52:27]
> On Thu, 2006-12-07 at 16:38 +0100, Jacob Gorm Hansen wrote:
>> >From dom0 I can ping and TCP connect to the domain just fine, but from
>> the domU the TCP connection hangs after the first few packets. UIP
>> complains about missing TCP checksums on the incoming packets,
>
> You can get round that one by writing a feature-no-csum-offload=1 node
> in xenstore. netfront.c in the sparse tree does this conditionally in
> order to support older Linux kernels that also don't cope well with
> missing checksums. With that set dom0 should do checksumming for you.

I worry about what seems like a lack of clarity around what checksum
offload capability is assumed by a domain about its' peer.

Let's imagine a dom0 kernel that is minimised - it has as much
non-essential stuff removed as possible.  That would include any
protocol support that is not necessary for its functioning as a dom0.

A domU runs on top of the dom0 and wishes to use a protocol that is
not present in the dom0 kernel (SCTP, perhaps).

If the domU kernel produces an SCTP packet and doesn't bother to
calculate the relevant checksum, on the basis that dom0 is providing
that facility (either via hardware in the NIC or software), what
should dom0 do when it doesn't recognise the SCTP packet?

Perhaps the details of checksum offload capability need to be more
precisely specified (using feature declarations in the store).

The mechanism introduced recently to allow a domU to say "please don't
send me NETTXF_csum_blank packets" is useful - it would also be
convenient to allow the dom0 to make the same request of any domU.
(If a patch for this would be accepted I'll do it.)

dme.
-- 
David Edmondson, Sun Microsystems, http://www.dme.org

Re: Re: Trouble with TCP between domUs

[Herbert Xu]

David Edmondson <dme@sun.com> wrote:
> 
> I worry about what seems like a lack of clarity around what checksum
> offload capability is assumed by a domain about its' peer.

In Linux we support protocol-neutral checksum offload.  This works
whereby the guest simply sets an offset where we should store the
final checksum and an offset where the checksum should be computed
from.

This way the dom0 operating system can always fill in for NICs that
don't support all protocols.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: Trouble with TCP between domUs

[David Edmondson]

* herbert@gondor.apana.org.au [2006-12-08 05:19:46]
> David Edmondson <dme@sun.com> wrote:
>> 
>> I worry about what seems like a lack of clarity around what checksum
>> offload capability is assumed by a domain about its' peer.
>
> In Linux we support protocol-neutral checksum offload.  This works
> whereby the guest simply sets an offset where we should store the
> final checksum and an offset where the checksum should be computed
> from.

The inter-domain protocol has no facility to allow the guest to
specify the offset at which the checksum should be placed - how does
the peer know where to put it?

And "Protocol neutral" means "as long as the checksum algorithm used
is the IP one" :-)

dme.
-- 
David Edmondson, Sun Microsystems, http://www.dme.org

Re: Re: Trouble with TCP between domUs

[Herbert Xu]

David Edmondson <dme@sun.com> wrote:
> 
> The inter-domain protocol has no facility to allow the guest to
> specify the offset at which the checksum should be placed - how does
> the peer know where to put it?

When I extended the netback protocol for TSO the idea was that you
can also have an extra descriptor for checksum offload that would
carry the offsets.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: Trouble with TCP between domUs

[David Edmondson]

* herbert@gondor.apana.org.au [2006-12-09 01:13:26]
> David Edmondson <dme@sun.com> wrote:
>> The inter-domain protocol has no facility to allow the guest to
>> specify the offset at which the checksum should be placed - how does
>> the peer know where to put it?
>
> When I extended the netback protocol for TSO the idea was that you
> can also have an extra descriptor for checksum offload that would
> carry the offsets.

But that offset is not present today, suggesting that the peer
receiving the packet must be parsing it to determine where to place
the checksum (though I couldn't find that code in a quick inspection).

If the receiver of the packet is required to parse it we're back at
the problem I originally described.

The GSO feature flags are quite specific about what is supported -
checksum feature flags should be the same.

dme.
-- 
David Edmondson, Sun Microsystems, http://www.dme.org

Re: Trouble with TCP between domUs

[David Edmondson]

* herbert@gondor.apana.org.au [2006-12-09 01:13:26]
> David Edmondson <dme@sun.com> wrote:
>> The inter-domain protocol has no facility to allow the guest to
>> specify the offset at which the checksum should be placed - how does
>> the peer know where to put it?
>
> When I extended the netback protocol for TSO the idea was that you
> can also have an extra descriptor for checksum offload that would
> carry the offsets.

But that offset is not present today, suggesting that the peer
receiving the packet must be parsing it to determine where to place
the checksum (though I couldn't find that code in a quick inspection).

If the receiver of the packet is required to parse it we're back at
the problem I originally described.

The GSO feature flags are quite specific about what is supported -
checksum feature flags should be the same.

dme.
-- 
David Edmondson, Sun Microsystems, http://www.dme.org

Re: Re: Trouble with TCP between domUs

[Herbert Xu]

David Edmondson <dme@sun.com> wrote:
>
>> When I extended the netback protocol for TSO the idea was that you
>> can also have an extra descriptor for checksum offload that would
>> carry the offsets.
> 
> But that offset is not present today, suggesting that the peer
> receiving the packet must be parsing it to determine where to place
> the checksum (though I couldn't find that code in a quick inspection).

What I meant is that the checksum offload can use the same mechanism
as GSO to pass the offset in a descriptor.  This has not been implemented
yet.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt