[uml-devel] uml "ip header error" for large (fragmented) udp packets

[uml-devel] uml "ip header error" for large (fragmented) udp packets

[Lars Ellenberg]

in the upcomming new heartbeat design of linux-ha, we use relatively
large udp packets (passing large xml blobs back and forth between the
nodes). now, cluster development and testing is convenient on UML.

but:
 uml seems to silently and systematically lose all fragmented
 udp packets, that is packages larger than mtu, and 1480 byte
 max for xml blobs just does not work out too good ...

I investigated a little bit (using two simple perl snippets to
generate/[not]receive the udp packets), and it turns out that:

 sending large (up to 64K; fragmented) udp packets
 using tuntap:
  HOST -> UML  works.
  UML  -> HOST nope :(
  UML <-> UML  nope :( [ neither with mcast or other ]
 
 looking into /proc/net/snmp on the UMLs and the HOST
 show on the not receiving side an increase of
 InHdrError!
 (it never is reassembled into a proper udp packet)
 
this is easy to reproduce (because it just happens all the time)
tested with 2.6.6 and 2.6.8 plus respective UML patches.

it seems to me that UML corrupts the ip header of
fragmented udp packets somehow at sending time.

I wonder, if someone uses nfs over udp on uml, this should
be a long known issue and turn up loads of hits in a search.
I did not find a single reference to that problem, though.

the way it is now we probably need to reanimate some of our old
boxes to form a real test cluster. and believe me, that is no fun :(

if some kind soul would be able to fix that...
would make cluster testing as we do it
so much more convenient :-)

thanks,

	Lars Ellenberg

please CC me, I'm not subscribed on this list.


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] uml "ip header error" for large (fragmented) udp packets

[BlaisorBlade]

On Sunday 03 October 2004 04:36, Lars Ellenberg wrote:
> in the upcomming new heartbeat design of linux-ha, we use relatively
> large udp packets (passing large xml blobs back and forth between the
> nodes). now, cluster development and testing is convenient on UML.
>
> but:
>  uml seems to silently and systematically lose all fragmented
>  udp packets, that is packages larger than mtu, and 1480 byte
>  max for xml blobs just does not work out too good ...

> I investigated a little bit (using two simple perl snippets to
> generate/[not]receive the udp packets), and it turns out that:

>  sending large (up to 64K; fragmented) udp packets
>  using tuntap:
>   HOST -> UML  works.
>   UML  -> HOST nope :(
>   UML <-> UML  nope :( [ neither with mcast or other ]
>
>  looking into /proc/net/snmp on the UMLs and the HOST
>  show on the not receiving side an increase of
>  InHdrError!
>  (it never is reassembled into a proper udp packet)
>
> this is easy to reproduce (because it just happens all the time)
> tested with 2.6.6 and 2.6.8 plus respective UML patches.

Have you tried to use different UML transports (ethertap/uml_switch)- 
tcpdump'ing the traffic - other things? Changing the host version? Increasing 
the MTU somewhere (but seems not to work)? What is strange is that the UML 
code does not even parse the IP header when using tuntap - it only works at 
the Ethernet layer. IIRC, the fragmentation happens at the IP layer...

Also, can you post the scripts you use?

> it seems to me that UML corrupts the ip header of
> fragmented udp packets somehow at sending time.
>
> I wonder, if someone uses nfs over udp on uml, this should
> be a long known issue and turn up loads of hits in a search.
> I did not find a single reference to that problem, though.

Does NFS uses large UDP packets?

> thanks,
>
> 	Lars Ellenberg
>
> please CC me, I'm not subscribed on this list.

-- 
Paolo Giarrusso, aka Blaisorblade
Linux registered user n. 292729


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] uml "ip header error" for large (fragmented) udp packets

[Lars Ellenberg]

[-- Attachment #1: Type: text/plain, Size: 2762 bytes --]

/ 2004-10-03 17:11:26 +0200
\ BlaisorBlade:
> On Sunday 03 October 2004 04:36, Lars Ellenberg wrote:
> > in the upcomming new heartbeat design of linux-ha, we use relatively
> > large udp packets (passing large xml blobs back and forth between the
> > nodes). now, cluster development and testing is convenient on UML.
> >
> > but:
> >  uml seems to silently and systematically lose all fragmented
> >  udp packets, that is packages larger than mtu, and 1480 byte
> >  max for xml blobs just does not work out too good ...
> 
> > I investigated a little bit (using two simple perl snippets to
> > generate/[not]receive the udp packets), and it turns out that:
> 
> >  sending large (up to 64K; fragmented) udp packets
> >  using tuntap:
> >   HOST -> UML  works.
> >   UML  -> HOST nope :(
> >   UML <-> UML  nope :( [ neither with mcast or other ]
> >
> >  looking into /proc/net/snmp on the UMLs and the HOST
> >  show on the not receiving side an increase of
> >  InHdrError!
> >  (it never is reassembled into a proper udp packet)
> >
> > this is easy to reproduce (because it just happens all the time)
> > tested with 2.6.6 and 2.6.8 plus respective UML patches.
> 
> Have you tried to use different UML transports (ethertap/uml_switch)- 

tuntap with uml_switch, uml_switch -hub
multicast

> tcpdump'ing the traffic - other things?

host tcpdump sees packets, they appear to be correct.

tcpdump in the uml segfaults
on the second fragment for two-fragment udp packet :-) 

> Changing the host version?

host linux kernel:
2.4.21-suse something, 2.4.26, 2.6.6, 2.6.8

> Increasing the MTU somewhere (but seems not to work)?

right. does not work.

> What is strange is that the UML code does not even parse the IP header
> when using tuntap - it only works at the Ethernet layer.
> IIRC, the fragmentation happens at the IP layer...

hm... 
I only describe what I see.

> Also, can you post the scripts you use?

attached.
on one side: perl UdpRecv.pl &
on other uml, host, or same system, does not matter:
perl UdpSend.pl [<target ip>]

btw, uml to itself via uml-lo does work...

> > it seems to me that UML corrupts the ip header of
> > fragmented udp packets somehow at sending time.
> >
> > I wonder, if someone uses nfs over udp on uml, this should
> > be a long known issue and turn up loads of hits in a search.
> > I did not find a single reference to that problem, though.
> 
> Does NFS uses large UDP packets?

sometimes.
and yes, I just exported from uml,
nfs-mounted on host, did an ls in a directory with MANY files,
and it never came back
(udp never reached host, host increases InHdrError)
blocksize of exported file system was 1024, mtu is 1500, on would assume
it should just work ... but that is still an other problem.


thanks,

	lge

[-- Attachment #2: UdpRecv.pl --]
[-- Type: application/x-perl, Size: 973 bytes --]

[-- Attachment #3: UdpSend.pl --]
[-- Type: application/x-perl, Size: 1178 bytes --]

Re: [uml-devel] uml "ip header error" for large (fragmented) udp packets

[Gerd Knorr]

BlaisorBlade <blaisorblade_spam@yahoo.it> writes:

> > I wonder, if someone uses nfs over udp on uml, this should
> > be a long known issue and turn up loads of hits in a search.
> > I did not find a single reference to that problem, though.
> 
> Does NFS uses large UDP packets?

Looks like it does.  I can ack that issue for the NFS case.  NFS over
tcp does fine (which seems to be the default, so I didn't notice until
now).  NFS over udp works, is very slow through and I get plenty of
"nfs server not responding" + "nfs server ok" messages in the syslog.
Looks like it doesn't loose all packets, but enougth to slowdown it
drastically and trigger timeouts on the client side.

That is (kernel) nfs server on the host machine, uml being connected
via tuntap networking and mounting /home using NFS.

  Gerd

-- 
return -ENOSIG;


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] uml "ip header error" for large (fragmented) udp packets

[Henrik Nordstrom]

On Mon, 4 Oct 2004, Gerd Knorr wrote:

>> Does NFS uses large UDP packets?
>
> Looks like it does.

The NFS message size is normally 4096 + message headers + protocol layers. 
The rsize/wsize mount parameters has a play in this.. (NFS data payload 
size within the NFS RPC message over UDP).

Regards
Henrik


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] uml "ip header error" for large (fragmented) udp packets

[BlaisorBlade]

On Sunday 03 October 2004 04:36, Lars Ellenberg wrote:
> in the upcomming new heartbeat design of linux-ha, we use relatively
> large udp packets (passing large xml blobs back and forth between the
> nodes). now, cluster development and testing is convenient on UML.

> but:
>  uml seems to silently and systematically lose all fragmented
>  udp packets, that is packages larger than mtu, and 1480 byte
>  max for xml blobs just does not work out too good ...

> I investigated a little bit (using two simple perl snippets to
> generate/[not]receive the udp packets), and it turns out that:

>  sending large (up to 64K; fragmented) udp packets
>  using tuntap:
>   HOST -> UML  works.
>   UML  -> HOST nope :(
>   UML <-> UML  nope :( [ neither with mcast or other ]
>
>  looking into /proc/net/snmp on the UMLs and the HOST
>  show on the not receiving side an increase of
>  InHdrError!
>  (it never is reassembled into a proper udp packet)
>
> this is easy to reproduce (because it just happens all the time)
> tested with 2.6.6 and 2.6.8 plus respective UML patches.
>
> it seems to me that UML corrupts the ip header of
> fragmented udp packets somehow at sending time.

I've traced this with Ethereal (v0.10.5) running on tap0 and it complains that 
the IP header checksum is always incorrect when the packet is fragmented. 
This does not happen when running both programs on the host; I've set an mtu 
of 1500 for "lo" fot this test.

However, it seems that Ethereal always shows the UDP checksum, which is 
different, as incorrect for not fragmented packets, when they are sent over 
the "lo" link (on my 2.6.7 host kernel); by comparison, when sending them 
over local network it never complains. The Ethereal doc say that when 
capturing on an interface that supports TCP checksum offloading (i.e. 
hardware checksumming), this is normal for TCP checksums, so I guess this can 
happen for UDP checksums, too.

But why the loopback driver should mark itself as capable of doing "hardware 
checksum"? However, it seems that actually this is the situation. In the 
source code, the loopback driver is marked as "needing no checksum at all 
because it's safe (see NETIF_F_NO_CSUM in include/linux/skbuff.h).

Also, it seems that the UML code happily ignores specifying what checksum 
support. And this could help us.

include/linux/skbuff.h describes the Checksum flags, and UML does not use 
them: these two commands return no output.

find arch/um/ -name '*.[ch]'|xargs grep NETIF
find arch/um/ -name '*.[ch]'|xargs grep CHECKSUM

Actually I've never done any work *at all* on the networking code, so this is 
just a wild guess.
> the way it is now we probably need to reanimate some of our old
> boxes to form a real test cluster. and believe me, that is no fun :(

I've tried UML 2.4, and it does not seem to experience this bug: it does not 
increases the host error count in /proc/net/snmp, UdpRecv receives all 
packet sizes (I stopped the test at 49100 bytes), and even Ethereal shows 
correct datas. The test were run sending 
the packets from UML to the Host, as you say.

So this could help you for now, while we try to find a clue about this. Quite 
frankly, I must say that I'm not seeing any network kernel hacker here 
(correct me if I'm wrong), so it will take some time to debug it. Maybe Gerd 
Knorr is an exception, actually.

> if some kind soul would be able to fix that...
> would make cluster testing as we do it
> so much more convenient :-)

-- 
Paolo Giarrusso, aka Blaisorblade
Linux registered user n. 292729


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] uml "ip header error" for large (fragmented) udp packets

[Lars Ellenberg]

> So this could help you for now, while we try to find a clue about this. Quite 
> frankly, I must say that I'm not seeing any network kernel hacker here 
> (correct me if I'm wrong), so it will take some time to debug it. Maybe Gerd 
> Knorr is an exception, actually.

Well, then I take Andi Kleen and Lars Marowsky-Brée into CC for now.
Lars, because I expect him to be interessted in having UML as full
featured cluster simulation tool available, and Andi because I hope he
might know the network code much better than me...

FYI, full thread can be found for example at
http://thread.gmane.org/gmane.linux.uml.devel/4607

Thanks,

	Lars Ellenberg

/ 2004-10-06 19:50:46 +0200
\ BlaisorBlade:
> On Sunday 03 October 2004 04:36, Lars Ellenberg wrote:
> > in the upcomming new heartbeat design of linux-ha, we use relatively
> > large udp packets (passing large xml blobs back and forth between the
> > nodes). now, cluster development and testing is convenient on UML.
> 
> > but:
> >  uml seems to silently and systematically lose all fragmented
> >  udp packets, that is packages larger than mtu, and 1480 byte
> >  max for xml blobs just does not work out too good ...
> 
> > I investigated a little bit (using two simple perl snippets to
> > generate/[not]receive the udp packets), and it turns out that:
> 
> >  sending large (up to 64K; fragmented) udp packets
> >  using tuntap:
> >   HOST -> UML  works.
> >   UML  -> HOST nope :(
> >   UML <-> UML  nope :( [ neither with mcast or other ]
> >
> >  looking into /proc/net/snmp on the UMLs and the HOST
> >  show on the not receiving side an increase of
> >  InHdrError!
> >  (it never is reassembled into a proper udp packet)
> >
> > this is easy to reproduce (because it just happens all the time)
> > tested with 2.6.6 and 2.6.8 plus respective UML patches.
> >
> > it seems to me that UML corrupts the ip header of
> > fragmented udp packets somehow at sending time.
> 
> I've traced this with Ethereal (v0.10.5) running on tap0 and it complains that 
> the IP header checksum is always incorrect when the packet is fragmented. 
> This does not happen when running both programs on the host; I've set an mtu 
> of 1500 for "lo" fot this test.
> 
> However, it seems that Ethereal always shows the UDP checksum, which is 
> different, as incorrect for not fragmented packets, when they are sent over 
> the "lo" link (on my 2.6.7 host kernel); by comparison, when sending them 
> over local network it never complains. The Ethereal doc say that when 
> capturing on an interface that supports TCP checksum offloading (i.e. 
> hardware checksumming), this is normal for TCP checksums, so I guess this can 
> happen for UDP checksums, too.
> 
> But why the loopback driver should mark itself as capable of doing "hardware 
> checksum"? However, it seems that actually this is the situation. In the 
> source code, the loopback driver is marked as "needing no checksum at all 
> because it's safe (see NETIF_F_NO_CSUM in include/linux/skbuff.h).
> 
> Also, it seems that the UML code happily ignores specifying what checksum 
> support. And this could help us.
> 
> include/linux/skbuff.h describes the Checksum flags, and UML does not use 
> them: these two commands return no output.
> 
> find arch/um/ -name '*.[ch]'|xargs grep NETIF
> find arch/um/ -name '*.[ch]'|xargs grep CHECKSUM
> 
> Actually I've never done any work *at all* on the networking code, so this is 
> just a wild guess.
> > the way it is now we probably need to reanimate some of our old
> > boxes to form a real test cluster. and believe me, that is no fun :(
> 
> I've tried UML 2.4, and it does not seem to experience this bug: it does not 
> increases the host error count in /proc/net/snmp, UdpRecv receives all 
> packet sizes (I stopped the test at 49100 bytes), and even Ethereal shows 
> correct datas. The test were run sending 
> the packets from UML to the Host, as you say.
> 
> So this could help you for now, while we try to find a clue about this. Quite 
> frankly, I must say that I'm not seeing any network kernel hacker here 
> (correct me if I'm wrong), so it will take some time to debug it. Maybe Gerd 
> Knorr is an exception, actually.
> 
> > if some kind soul would be able to fix that...
> > would make cluster testing as we do it
> > so much more convenient :-)
> 
> -- 
> Paolo Giarrusso, aka Blaisorblade
> Linux registered user n. 292729


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] uml "ip header error" for large (fragmented) udp packets

[Andi Kleen]

On Wed, Oct 06, 2004 at 08:48:23PM +0200, Lars Ellenberg wrote:
> > So this could help you for now, while we try to find a clue about this. Quite 
> > frankly, I must say that I'm not seeing any network kernel hacker here 
> > (correct me if I'm wrong), so it will take some time to debug it. Maybe Gerd 
> > Knorr is an exception, actually.
> 
> Well, then I take Andi Kleen and Lars Marowsky-Br?e into CC for now.
> Lars, because I expect him to be interessted in having UML as full
> featured cluster simulation tool available, and Andi because I hope he
> might know the network code much better than me...
> 
> FYI, full thread can be found for example at
> http://thread.gmane.org/gmane.linux.uml.devel/4607

Paolo's analysis is basically correct. loopback sets this flag
for better performance.  Actually in 2.6 it probably doesn't help
very much anymore because TCP can do checksum copy RX now, and that
would get the checksum basically for free. But it's still there
and may still make things slightly faster.

If UML taps the packets from lo it will see incorrect checksums.

Using a tun or ethertap device would avoid this. In the worst
case you could also just delete the flag from the loopback
interface, it's only an optimization.

-Andi



-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] uml "ip header error" for large (fragmented) udp packets

[Lars Ellenberg]

/ 2004-10-06 22:35:39 +0200
\ Andi Kleen:
> On Wed, Oct 06, 2004 at 08:48:23PM +0200, Lars Ellenberg wrote:
> > > So this could help you for now, while we try to find a clue about this. Quite 
> > > frankly, I must say that I'm not seeing any network kernel hacker here 
> > > (correct me if I'm wrong), so it will take some time to debug it. Maybe Gerd 
> > > Knorr is an exception, actually.
> > 
> > Well, then I take Andi Kleen and Lars Marowsky-Br?e into CC for now.
> > Lars, because I expect him to be interessted in having UML as full
> > featured cluster simulation tool available, and Andi because I hope he
> > might know the network code much better than me...
> > 
> > FYI, full thread can be found for example at
> > http://thread.gmane.org/gmane.linux.uml.devel/4607
> 
> Paolo's analysis is basically correct. loopback sets this flag
> for better performance.  Actually in 2.6 it probably doesn't help
> very much anymore because TCP can do checksum copy RX now, and that
> would get the checksum basically for free. But it's still there
> and may still make things slightly faster.
> 
> If UML taps the packets from lo it will see incorrect checksums.
> 
> Using a tun or ethertap device would avoid this. In the worst
> case you could also just delete the flag from the loopback
> interface, it's only an optimization.
> 
> -Andi

unfortunately ethertap transport does not work either,
at least if UML is 2.6.6 and host kernel is 2.4.21-suse-whatever...
I did not try other combinations yet, but I doubt that changes a thing.

you suggest that we remove NET_IF_F_NO_CSUM from lo in the host?
ok, I'll try recompile my host then, and followup if that helps.

	lge



-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] uml "ip header error" for large (fragmented) udp packets

[BlaisorBlade]

On Wednesday 06 October 2004 23:48, Lars Ellenberg wrote:
> / 2004-10-06 22:35:39 +0200
>
> \ Andi Kleen:
> > On Wed, Oct 06, 2004 at 08:48:23PM +0200, Lars Ellenberg wrote:
> > > > So this could help you for now, while we try to find a clue about
> > > > this. Quite frankly, I must say that I'm not seeing any network
> > > > kernel hacker here (correct me if I'm wrong), so it will take some
> > > > time to debug it. Maybe Gerd Knorr is an exception, actually.
> > >
> > > Well, then I take Andi Kleen and Lars Marowsky-Br?e into CC for now.
> > > Lars, because I expect him to be interessted in having UML as full
> > > featured cluster simulation tool available, and Andi because I hope he
> > > might know the network code much better than me...
> > >
> > > FYI, full thread can be found for example at
> > > http://thread.gmane.org/gmane.linux.uml.devel/4607
> >
> > Paolo's analysis is basically correct. loopback sets this flag
> > for better performance.  Actually in 2.6 it probably doesn't help
> > very much anymore because TCP can do checksum copy RX now, and that
> > would get the checksum basically for free. But it's still there
> > and may still make things slightly faster.
> >
> > If UML taps the packets from lo it will see incorrect checksums.
It does not, so the solution is not the right one.
> > Using a tun or ethertap device would avoid this. In the worst
> > case you could also just delete the flag from the loopback
> > interface, it's only an optimization.
> >
> > -Andi
>
> unfortunately ethertap transport does not work either,
> at least if UML is 2.6.6 and host kernel is 2.4.21-suse-whatever...
> I did not try other combinations yet, but I doubt that changes a thing.
>
> you suggest that we remove NET_IF_F_NO_CSUM from lo in the host?
> ok, I'll try recompile my host then, and followup if that helps.

No, I think he spoke about the guest; also he misunderstood a bit the problem, 
since the packets do not go through the "lo" interface inside UML.

-- 
Paolo Giarrusso, aka Blaisorblade
Linux registered user n. 292729


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] uml "ip header error" for large (fragmented) udp packets

[BlaisorBlade]

On Wednesday 06 October 2004 22:35, Andi Kleen wrote:
> On Wed, Oct 06, 2004 at 08:48:23PM +0200, Lars Ellenberg wrote:
> > > So this could help you for now, while we try to find a clue about this.
> > > Quite frankly, I must say that I'm not seeing any network kernel hacker
> > > here (correct me if I'm wrong), so it will take some time to debug it.
> > > Maybe Gerd Knorr is an exception, actually.

> > Well, then I take Andi Kleen and Lars Marowsky-Br?e into CC for now.
> > Lars, because I expect him to be interessted in having UML as full
> > featured cluster simulation tool available, and Andi because I hope he
> > might know the network code much better than me...

> > FYI, full thread can be found for example at
> > http://thread.gmane.org/gmane.linux.uml.devel/4607

> Paolo's analysis is basically correct. loopback sets this flag
> for better performance.  Actually in 2.6 it probably doesn't help
> very much anymore because TCP can do checksum copy RX now, and that
> would get the checksum basically for free. But it's still there
> and may still make things slightly faster.
First thing: thanks a lot for your quick answer.

My discussion about "lo" was slightly unrelated to the exact problem, and a 
bit confusing...

I was at first surprised from Ethereal complaining about the host kernel, so I 
thought I could have a buggy Ethereal, and then went checking that it's a 
Linux optimization, indeed.

> If UML taps the packets from lo it will see incorrect checksums.
> Using a tun or ethertap device would avoid this.

> In the worst 
> case you could also just delete the flag from the loopback
> interface, it's only an optimization.

No, inside the Uml kernel they go through a virtual "ethN" interface, which 
uses special code. That driver, in turn, will use either ethertap, or TAP (it 
sends Ethernet frames), or even other mechanism.

You can find it (in 2.6.9-rc2 at least) in arch/um/drivers/net_*.c and 
arch/um/os-Linux/drivers/*tap*.c. The code in *_kern.c files links against 
the kernel API and includes, *_user.c against the host userspace includes.

And the problem is, probably, that the UML network drivers never declare their 
checksumming status, as I said in the previous mail:

[quote]
include/linux/skbuff.h describes the Checksum flags, and UML does not use 
them: these two commands return no (relevant) output.

find arch/um/ -name '*.[ch]'|xargs grep NETIF
find arch/um/ -name '*.[ch]'|xargs grep CHECKSUM
[/quote]

And not even these ones:

find arch/um/ -name '*.[ch]'|xargs grep NETIF
find arch/um/ -name '*.[ch]'|xargs grep CHECKSUM

Also, it's possible that there are even other bugs...
-- 
Paolo Giarrusso, aka Blaisorblade
Linux registered user n. 292729


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] uml "ip header error" for large (fragmented) udp packets

[Lars Ellenberg]

/ 2004-10-07 20:41:51 +0200
\ BlaisorBlade:
> On Wednesday 06 October 2004 22:35, Andi Kleen wrote:
> > On Wed, Oct 06, 2004 at 08:48:23PM +0200, Lars Ellenberg wrote:

> > > FYI, full thread can be found for example at
> > > http://thread.gmane.org/gmane.linux.uml.devel/4607
> 
> > Paolo's analysis is basically correct. loopback sets this flag
> > for better performance.  Actually in 2.6 it probably doesn't help
> > very much anymore because TCP can do checksum copy RX now, and that
> > would get the checksum basically for free. But it's still there
> > and may still make things slightly faster.
> First thing: thanks a lot for your quick answer.
> 
> My discussion about "lo" was slightly unrelated to the exact problem, and a 
> bit confusing...
> 
> I was at first surprised from Ethereal complaining about the host kernel, so I 
> thought I could have a buggy Ethereal, and then went checking that it's a 
> Linux optimization, indeed.
> 
> > If UML taps the packets from lo it will see incorrect checksums.
> > Using a tun or ethertap device would avoid this.
> 
> > In the worst 
> > case you could also just delete the flag from the loopback
> > interface, it's only an optimization.
> 
> No, inside the Uml kernel they go through a virtual "ethN" interface, which 
> uses special code. That driver, in turn, will use either ethertap, or TAP (it 
> sends Ethernet frames), or even other mechanism.
> 
> You can find it (in 2.6.9-rc2 at least) in arch/um/drivers/net_*.c and 
> arch/um/os-Linux/drivers/*tap*.c. The code in *_kern.c files links against 
> the kernel API and includes, *_user.c against the host userspace includes.
> 
> And the problem is, probably, that the UML network drivers never declare their 
> checksumming status, as I said in the previous mail:
> 
> [quote]
> include/linux/skbuff.h describes the Checksum flags, and UML does not use 
> them: these two commands return no (relevant) output.
> 
> find arch/um/ -name '*.[ch]'|xargs grep NETIF
> find arch/um/ -name '*.[ch]'|xargs grep CHECKSUM
> [/quote]
> 
> And not even these ones:
> 
> find arch/um/ -name '*.[ch]'|xargs grep NETIF
> find arch/um/ -name '*.[ch]'|xargs grep CHECKSUM
> 
> Also, it's possible that there are even other bugs...

now, what I found:
arch/um/drivers/net_kern.c:
struct sk_buff *ether_adjust_skb(struct sk_buff *skb, int extra)
{
	if((skb != NULL) && (skb_tailroom(skb) < extra)){
	  	struct sk_buff *skb2;

		skb2 = skb_copy_expand(skb, 0, extra, GFP_ATOMIC);
		dev_kfree_skb(skb);
		skb = skb2;
	}
	if(skb != NULL) skb_put(skb, extra);
	return(skb);
}

net/core/skbuff.c:
 *	BUG ALERT: ip_summed is not copied. Why does this work? Is it used
 *	only by netfilter in the cases when checksum is recalculated? --ANK
 */
struct sk_buff *skb_copy_expand(const struct sk_buff *skb,
				int newheadroom, int newtailroom, int gfp_mask)
{


does that trigger something in someones brain maybe?
someone "sees" it?

otherwise I keep poking around...

	lge



-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] uml "ip header error" for large (fragmented) udp packets

[Lars Ellenberg]

/ 2004-10-07 22:24:54 +0200
\ Lars Ellenberg:

> now, what I found:
> arch/um/drivers/net_kern.c:
> struct sk_buff *ether_adjust_skb(struct sk_buff *skb, int extra)
> {
> 	if((skb != NULL) && (skb_tailroom(skb) < extra)){
> 	  	struct sk_buff *skb2;
> 
> 		skb2 = skb_copy_expand(skb, 0, extra, GFP_ATOMIC);
> 		dev_kfree_skb(skb);
> 		skb = skb2;
> 	}
> 	if(skb != NULL) skb_put(skb, extra);
> 	return(skb);
> }
> 
> net/core/skbuff.c:
>  *	BUG ALERT: ip_summed is not copied. Why does this work? Is it used
>  *	only by netfilter in the cases when checksum is recalculated? --ANK
>  */
> struct sk_buff *skb_copy_expand(const struct sk_buff *skb,
> 				int newheadroom, int newtailroom, int gfp_mask)
> {
> 
> 
> does that trigger something in someones brain maybe?
> someone "sees" it?
> 
> otherwise I keep poking around...

since UML => Host does not work, but
Host => UML does, this suggests that
the bug is somewhere on the sending side.

I was not able to track it down.

but, I just patched out the "verify checksum" from the receiving part,
and get 16000 byte through now, sometimes more (this seems to be a
buffer issue).

so for now, I just don't care for the ip checksum on my UMLs,
and I have to live with no fragmented UDP from UML => Host.
but between my UMLs, I have now up to 16k UDP, that should be enough for
the moment beeing.


bute but effective: dont care for checksums.  at first I had additional
printks in the code wherever it said "goto inhdr_error", to find where
exactly it breaks.
only the now #if 0'ed one triggered.

	Lars Ellenberg


--- linux-2.6.6/net/ipv4/ip_input.c.orig	2004-10-11 19:35:57.000000000 +0200
+++ linux-2.6.6/net/ipv4/ip_input.c	2004-10-11 19:53:58.000000000 +0200
@@ -403,8 +403,10 @@
 
 	iph = skb->nh.iph;
 
+#if 0
 	if (ip_fast_csum((u8 *)iph, iph->ihl) != 0)
 		goto inhdr_error; 
+#endif
 
 	{
 		__u32 len = ntohs(iph->tot_len); 


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] SOLVED: uml "ip header error" for large (fragmented) udp packets

[Lars Ellenberg]

/ 2004-10-11 19:55:12 +0200
\ Lars Ellenberg:
> since UML => Host does not work, but
> Host => UML does, this suggests that
> the bug is somewhere on the sending side.
> 
> I was not able to track it down.

just so you know, finally:

=======================
--- linux-2.6.6/arch/um/include/sysdep-i386/checksum.h.orig	2004-10-12 01:50:49.000000000 +0200
+++ linux-2.6.6/arch/um/include/sysdep-i386/checksum.h	2004-10-12 01:50:58.000000000 +0200
@@ -102,8 +102,7 @@
 	   are modified, we must also specify them as outputs, or gcc
 	   will assume they contain their original values. */
 	: "=r" (sum), "=r" (iph), "=r" (ihl)
-	: "1" (iph), "2" (ihl)
-	: "memory");
+	: "1" (iph), "2" (ihl));
 	return(sum);
 }
 
=======================

that's all, folks. only a missing memory barrier.

WTF :-/

same patch applies to 2.6.8.1 - uml and probably all other umls.
since that is a one-to-one copy anyways, maybe UML should better use the
original (in include/asm-i386/checksum.h) right away??
there may be similar bugs hiding in various areas of uml...

Thanks for now,
keep it going...

btw,
anyone wants to give me a hint how to tune it best?
maybe how to up the mtu of the UML "nics"?

	Lars Ellenberg

"NLRge your UML-UDP" ...


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] SOLVED: uml "ip header error" for large (fragmented) udp packets

[Andi Kleen]

>  	   will assume they contain their original values. */
>  	: "=r" (sum), "=r" (iph), "=r" (ihl)
> -	: "1" (iph), "2" (ihl)
> -	: "memory");
> +	: "1" (iph), "2" (ihl));
>  	return(sum);
>  }
>  
> =======================

That's reverted, right? 
> 
> that's all, folks. only a missing memory barrier.
> 
> WTF :-/

This was fixed in mainline some time ago (several months probably more) 
The problem only started with newer gccs that optimize more aggressively.

> original (in include/asm-i386/checksum.h) right away??
> there may be similar bugs hiding in various areas of uml...

Sounds like a good idea. 

> 
> Thanks for now,
> keep it going...
> 
> btw,
> anyone wants to give me a hint how to tune it best?
> maybe how to up the mtu of the UML "nics"?

Don't go over 4K because the VM doesn't like >order 0 allocations
very much. But in general bigger is better.

-Andi


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] SOLVED: uml "ip header error" for large (fragmented) udp packets

[BlaisorBlade]

On Tuesday 12 October 2004 02:11, Andi Kleen wrote:
> >  	   will assume they contain their original values. */
> >
> >  	: "=r" (sum), "=r" (iph), "=r" (ihl)
> >
> > -	: "1" (iph), "2" (ihl)
> > -	: "memory");
> > +	: "1" (iph), "2" (ihl));
> >  	return(sum);
> >  }
> >
> > =======================
>
> That's reverted, right?
>
> > that's all, folks. only a missing memory barrier.
> >
> > WTF :-/
>
> This was fixed in mainline some time ago (several months probably more)
> The problem only started with newer gccs that optimize more aggressively.
>
> > original (in include/asm-i386/checksum.h) right away??
> > there may be similar bugs hiding in various areas of uml...
>
> Sounds like a good idea.
Agreed, but I've to check if the include does not have any conflict. And I 
don't have the time until after 2.6.9, because I must address more urgent 
issues. Obviously the one-liner itself is being sent to Andrew Morton.

> > Thanks for now,
> > keep it going...
> >
> > btw,
> > anyone wants to give me a hint how to tune it best?
> > maybe how to up the mtu of the UML "nics"?
>
> Don't go over 4K because the VM doesn't like >order 0 allocations
> very much. But in general bigger is better.

Sadly there is a problem: since we use TAP and emulate whole Ethernet frames, 
the code does not allow to increase the MTU to > 1500 bytes. I think this 
cannot be fixed currently, but if you think this is wrong, please let us now.

Obviously we could add another interface emulation with bigger MTU, but I've 
no ideas about which ones to emulate.
-- 
Paolo Giarrusso, aka Blaisorblade
Linux registered user n. 292729


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] SOLVED: uml "ip header error" for large (fragmented) udp packets

[Michael Richardson]

-----BEGIN PGP SIGNED MESSAGE-----


>>>>> "BlaisorBlade" == BlaisorBlade  <blaisorblade_spam@yahoo.it> writes:
    BlaisorBlade> Sadly there is a problem: since we use TAP and emulate
    BlaisorBlade> whole Ethernet frames, the code does not allow to
    BlaisorBlade> increase the MTU to > 1500 bytes. I think this cannot
    BlaisorBlade> be fixed currently, but if you think this is wrong,
    BlaisorBlade> please let us now.

  I'm certain that we could go as big as 2047 without a problem.
  It won't work when communicating outside the host, mind you.

  Beyond 2048, we have to use 802 SNAP encapsulation. I think that Linux
is jumbo ethernet ready, but I've never been there, so I don't know what
might be involved.

- --
]     "Elmo went to the wrong fundraiser" - The Simpson         |  firewalls  [
]   Michael Richardson,    Xelerance Corporation, Ottawa, ON    |net architect[
] mcr@xelerance.com      http://www.sandelman.ottawa.on.ca/mcr/ |device driver[
] panic("Just another Debian GNU/Linux using, kernel hacking, security guy"); [
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (GNU/Linux)
Comment: Finger me for keys

iQCVAwUBQWs5LYqHRg3pndX9AQFPpQP7Bj9cO0Mbm6/Ai0YHJ3bCXMSVko+IvIlq
GF1lFDQx6TEPtFtsSm0qMf8kvOnHV0+oJ/M21e1lSnKsgzltjtjs+1sGB8RKzzJi
iCn6mXkfEGQhc2oGE4isKUmGrMxZPVhXiwq/1MYAV+1ONuCPG77v7UMqD9fG5GOP
XmDV4uTykio=
=MJ+I
-----END PGP SIGNATURE-----


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] larger MTU _does_ work

[Lars Ellenberg]

/ 2004-10-12 03:10:59 +0200
\ BlaisorBlade:
> > > btw,
> > > anyone wants to give me a hint how to tune it best?
> > > maybe how to up the mtu of the UML "nics"?
> >
> > Don't go over 4K because the VM doesn't like >order 0 allocations
> > very much. But in general bigger is better.
> 
> Sadly there is a problem: since we use TAP and emulate whole Ethernet frames, 
> the code does not allow to increase the MTU to > 1500 bytes. I think this 
> cannot be fixed currently, but if you think this is wrong, please let us now.
> 
> Obviously we could add another interface emulation with bigger MTU, but I've 
> no ideas about which ones to emulate.

now, if I
  config eth0=daemon,FE:FD:00:00:00:01,,/tmp/uml0.ctl
and then
  ifconfig eth0 mtu 8192, uml does not complain
  (but does not work, either).

and if I then patch uml_switch from uml_utilities_20040114 (which is
what I have here, did not check whether there is anything newer out there)

===================
diff -ru tools.orig/uml_router/port.c tools/uml_router/port.c
--- tools.orig/uml_router/port.c	2003-03-12 16:19:03.000000000 +0100
+++ tools/uml_router/port.c	2004-10-15 16:59:07.000000000 +0200
@@ -14,7 +14,7 @@
     unsigned char src[ETH_ALEN];
     unsigned char proto[2];
   } header;
-  unsigned char data[1500];
+  unsigned char data[9000];
 };
 
 struct port {
===================

it even works!

I guess with mcast or other non-daemon transport (not yet tried),
it will just work, too (maybe you need to adjust the mtu on the host).

for the first time I really get 65507 byte udp packages through uml <-> uml.
(not that it makes any sense to use udp with that large messages ...)
anything larger won't work anyways...


	Lars Ellenberg



-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] SOLVED: uml "ip header error" for large (fragmented) udp packets

[BlaisorBlade]

On Tuesday 12 October 2004 02:03, Lars Ellenberg wrote:
> / 2004-10-11 19:55:12 +0200
>
> \ Lars Ellenberg:
> > since UML => Host does not work, but
> > Host => UML does, this suggests that
> > the bug is somewhere on the sending side.
> >
> > I was not able to track it down.
>
> just so you know, finally:

> =======================
> --- linux-2.6.6/arch/um/include/sysdep-i386/checksum.h.orig	2004-10-12
> 01:50:49.000000000 +0200 +++
> linux-2.6.6/arch/um/include/sysdep-i386/checksum.h	2004-10-12
> 01:50:58.000000000 +0200 @@ -102,8 +102,7 @@
>  	   are modified, we must also specify them as outputs, or gcc
>  	   will assume they contain their original values. */
>
>  	: "=r" (sum), "=r" (iph), "=r" (ihl)
>
> -	: "1" (iph), "2" (ihl)
> -	: "memory");
> +	: "1" (iph), "2" (ihl));
>  	return(sum);
>  }
The patch you posted REMOVES a memory barrier - you reversed it. I actually 
checked the barrier is missing in the source code; but the strange thing is 
that you modified checksum.h.orig and not checksum.h! Are you sure that you 
compiled the corrected header? However, the patch is correct, and I assume 
you posted it the right way. I'm going to send it to Andrew Morton for 2.6.9 
(hoping he wants to accept all these patches - they are rushing for 2.6.9).

> that's all, folks. only a missing memory barrier.
Thanks a lot, folks! Yes, the patch is right.
> WTF :-/
>
> same patch applies to 2.6.8.1 - uml and probably all other umls.
> since that is a one-to-one copy anyways, maybe UML should better use the
> original (in include/asm-i386/checksum.h) right away??
I could do this later - too much checks for the other header code for a 
one-minute patch.
> there may be similar bugs hiding in various areas of uml...
I think there are even worse ones.
> btw,
> anyone wants to give me a hint how to tune it best?
> maybe how to up the mtu of the UML "nics"?
Impossible with the current code - it emulates an ethernet card, so I don't 
think you can do anything for this. However I hope it does not cause too much 
problems. Search for the network howtos: someone explained how to set the 
interface packet scheduler (maybe even Jeff Dike on the main UML site).

Bye
-- 
Paolo Giarrusso, aka Blaisorblade
Linux registered user n. 292729


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] SOLVED: uml "ip header error" for large (fragmented) udp packets

[Lars Ellenberg]

/ 2004-10-12 02:27:00 +0200
\ BlaisorBlade:
> On Tuesday 12 October 2004 02:03, Lars Ellenberg wrote:
> > / 2004-10-11 19:55:12 +0200
> >
> > \ Lars Ellenberg:
> > > since UML => Host does not work, but
> > > Host => UML does, this suggests that
> > > the bug is somewhere on the sending side.
> > >
> > > I was not able to track it down.
> >
> > just so you know, finally:
> 
> > =======================
> > --- linux-2.6.6/arch/um/include/sysdep-i386/checksum.h.orig	2004-10-12
> > 01:50:49.000000000 +0200 +++
> > linux-2.6.6/arch/um/include/sysdep-i386/checksum.h	2004-10-12
> > 01:50:58.000000000 +0200 @@ -102,8 +102,7 @@
> >  	   are modified, we must also specify them as outputs, or gcc
> >  	   will assume they contain their original values. */
> >
> >  	: "=r" (sum), "=r" (iph), "=r" (ihl)
> >
> > -	: "1" (iph), "2" (ihl)
> > -	: "memory");
> > +	: "1" (iph), "2" (ihl));
> >  	return(sum);
> >  }
> The patch you posted REMOVES a memory barrier - you reversed it. I actually 
> checked the barrier is missing in the source code; but the strange thing is 
> that you modified checksum.h.orig and not checksum.h!

nope.
I just did "diff -u linux-2.6.6/arch/um/include/sysdep-i386/checksum.h{,.orig}"
instead of "diff -u linux-2.6.6/arch/um/include/sysdep-i386/checksum.h{.orig,}"
and did not notice.
Hey, it was in the middle of the night ;-)

thanks,
	lge


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel