[Qemu-devel] Re: [PATCH] networking using libpcap

[Qemu-devel] Re: [PATCH] networking using libpcap

[Sebastian Herbszt]

> I just discovered this patch
> 
> http://lists.freebsd.org/pipermail/freebsd-emulation/2007-February/003107.html
> 
> that implements network access using libpcap. Works perfect for me and 
> allows access to the local Ethernet right out of the box, very much 
> unlike tap and bridging. The attached version applies to trunk.

I have modified (e.g. got rid of threads) the original patch from the forum and
am using it here on Windows. It works fine but performance is pretty low.

> +ifdef CONFIG_PCAP
> +LIBS+=-lpcap
> +endif

On Windows it should be -lwpcap.

+    if ((fd = pcap_get_selectable_fd(s->handle)) < 0) {
+ fprintf(stderr, "qemu: pcap_get_selectable_fd failed\n");
+ goto fail;
+    }
+    qemu_set_fd_handler(fd, pcap_send, NULL, s);

pcap_get_selectable_fd() is not available on Windows. I just put
pcap_send() in main_loop_wait().

- Sebastian

[Qemu-devel] Re: [PATCH] networking using libpcap

[Jung-uk Kim]

> I just discovered this patch
> 
http://lists.freebsd.org/pipermail/freebsd-emulation/2007-February/003107.html
>
>     that implements network access using libpcap.

Since someone showed interest, I updated my patches against trunk. :-)

http://people.freebsd.org/~jkim/qemu-pcap-20080717.diff

I turned it off by default for now.  If you want to enable it, do:

configure --enable-pcap

> Works perfect for me and allows access to the local Ethernet right
> out of the box, very much unlike tap and bridging. The attached
> version applies to trunk.
> 
> I have modified (e.g. got rid of threads) the original patch from
> the forum and am using it here on Windows. It works fine but
> performance is pretty low.

*After* applying the new patch:

cp -p vl.c vl.c.orig
sed -e 's/#ifdef PCAP_SET_FILTER/#if 1/g' vl.c > vl.c.tmp
mv vl.c.tmp vl.c

and try again?  BTW, I have no real experience with WinPcap, so don't 
kill me if it does not work for you. ;-)
 
> +ifdef CONFIG_PCAP
> +LIBS+=-lpcap
> +endif
> 
> On Windows it should be -lwpcap.

Thanks for the tip!

> +    if ((fd = pcap_get_selectable_fd(s->handle)) < 0) {
> + fprintf(stderr, "qemu: pcap_get_selectable_fd failed\n");
> + goto fail;
> +    }
> +    qemu_set_fd_handler(fd, pcap_send, NULL, s);
> 
> pcap_get_selectable_fd() is not available on Windows. I just put
> pcap_send() in main_loop_wait().

I added WinPcap API support from WinPcap manual pages but I have no 
way of checking.  Can you try the patch and letting me know?

Juergen,

I made FreeBSD ports patch for emulators/qemu-devel:

http://people.freebsd.org/~jkim/qemu-devel-20080620-pcap.diff

FYI...

Thanks,

Jung-uk Kim

[Qemu-devel] Re: [PATCH] networking using libpcap

[Sebastian Herbszt]

Jung-uk Kim wrote:

> Since someone showed interest, I updated my patches against trunk. :-)
> 
> http://people.freebsd.org/~jkim/qemu-pcap-20080717.diff
> 
> I turned it off by default for now.  If you want to enable it, do:
> 
> configure --enable-pcap

To avoid confusion please change the pcap probe to error instead of
silently setting pcap=no. Take a look at audio_drv_probe.

>> Works perfect for me and allows access to the local Ethernet right
>> out of the box, very much unlike tap and bridging. The attached
>> version applies to trunk.
>> 
>> I have modified (e.g. got rid of threads) the original patch from
>> the forum and am using it here on Windows. It works fine but
>> performance is pretty low.
> 
> *After* applying the new patch:
> 
> cp -p vl.c vl.c.orig
> sed -e 's/#ifdef PCAP_SET_FILTER/#if 1/g' vl.c > vl.c.tmp
> mv vl.c.tmp vl.c
> 
> and try again?  BTW, I have no real experience with WinPcap, so don't 
> kill me if it does not work for you. ;-)

Using pcap_setfilter helps and colinux (conet-bridged-daemon) does use
one too. Currently your filter is
"ether dst 52:54:00:12:34:56 or ((broadcast or multicast) and not ether src 52:54:00:12:34:56)".
The filter used by colinux is
"(ether dst 00:ff:81:24:00:00) or (ether broadcast or multicast) or (ip broadcast or multicast)".

The "and not ether src 52:54:00:12:34:56" part in your filter prevents the VM from seeing own
packets. It doesn't reply to own "ping broadcast" where it does in colinux and VMware Server.

>> +ifdef CONFIG_PCAP
>> +LIBS+=-lpcap
>> +endif
>> 
>> On Windows it should be -lwpcap.
> 
> Thanks for the tip!
> 
>> +    if ((fd = pcap_get_selectable_fd(s->handle)) < 0) {
>> + fprintf(stderr, "qemu: pcap_get_selectable_fd failed\n");
>> + goto fail;
>> +    }
>> +    qemu_set_fd_handler(fd, pcap_send, NULL, s);
>> 
>> pcap_get_selectable_fd() is not available on Windows. I just put
>> pcap_send() in main_loop_wait().
> 
> I added WinPcap API support from WinPcap manual pages but I have no 
> way of checking.  Can you try the patch and letting me know?
 
Since you use the winpcap win32 only parts there is the following warning:

vl.c: In function `net_pcap_init':
vl.c:4247: warning: implicit declaration of function `pcap_getevent'
vl.c:4247: warning: assignment makes pointer from integer without a cast

You can avoid it by defining WPCAP in vl.c:

#if defined(CONFIG_PCAP)
#ifdef _WIN32
#define WPCAP 1
#endif
#include <pcap.h>
#endif

I noticed that if no ifname is passed you try to get one with pcap_lookupdev.
This is broken on winpcap, please see 
http://www.winpcap.org/pipermail/winpcap-bugs/2006-May/000220.html

I am not sure if pcap_lookupdev() gets you the "right" interface on non-win32,
so maybe just require ifname (tap does it too)?

Otherwise it seems to compile and run fine.

- Sebastian

[Qemu-devel] Re: [PATCH] networking using libpcap

[Jung-uk Kim]

On Friday 18 July 2008 12:12 pm, Sebastian Herbszt wrote:
> Jung-uk Kim wrote:
> > Since someone showed interest, I updated my patches against
> > trunk. :-)
> >
> > http://people.freebsd.org/~jkim/qemu-pcap-20080717.diff
> >
> > I turned it off by default for now.  If you want to enable it,
> > do:
> >
> > configure --enable-pcap
>
> To avoid confusion please change the pcap probe to error instead of
> silently setting pcap=no. Take a look at audio_drv_probe.

Done.

> >> Works perfect for me and allows access to the local Ethernet
> >> right out of the box, very much unlike tap and bridging. The
> >> attached version applies to trunk.
> >>
> >> I have modified (e.g. got rid of threads) the original patch
> >> from the forum and am using it here on Windows. It works fine
> >> but performance is pretty low.
> >
> > *After* applying the new patch:
> >
> > cp -p vl.c vl.c.orig
> > sed -e 's/#ifdef PCAP_SET_FILTER/#if 1/g' vl.c > vl.c.tmp
> > mv vl.c.tmp vl.c
> >
> > and try again?  BTW, I have no real experience with WinPcap, so
> > don't kill me if it does not work for you. ;-)
>
> Using pcap_setfilter helps and colinux (conet-bridged-daemon) does
> use one too. Currently your filter is
> "ether dst 52:54:00:12:34:56 or ((broadcast or multicast) and not
> ether src 52:54:00:12:34:56)". The filter used by colinux is
> "(ether dst 00:ff:81:24:00:00) or (ether broadcast or multicast) or
> (ip broadcast or multicast)".
>
> The "and not ether src 52:54:00:12:34:56" part in your filter
> prevents the VM from seeing own packets. It doesn't reply to own
> "ping broadcast" where it does in colinux and VMware Server.

I knew that I might have missed some edge cases. ;-P Corrected and 
enabled by default.

> >> +ifdef CONFIG_PCAP
> >> +LIBS+=-lpcap
> >> +endif
> >>
> >> On Windows it should be -lwpcap.
> >
> > Thanks for the tip!
> >
> >> +    if ((fd = pcap_get_selectable_fd(s->handle)) < 0) {
> >> + fprintf(stderr, "qemu: pcap_get_selectable_fd failed\n");
> >> + goto fail;
> >> +    }
> >> +    qemu_set_fd_handler(fd, pcap_send, NULL, s);
> >>
> >> pcap_get_selectable_fd() is not available on Windows. I just put
> >> pcap_send() in main_loop_wait().
> >
> > I added WinPcap API support from WinPcap manual pages but I have
> > no way of checking.  Can you try the patch and letting me know?
>
> Since you use the winpcap win32 only parts there is the following
> warning:
>
> vl.c: In function `net_pcap_init':
> vl.c:4247: warning: implicit declaration of function
> `pcap_getevent' vl.c:4247: warning: assignment makes pointer from
> integer without a cast
>
> You can avoid it by defining WPCAP in vl.c:
>
> #if defined(CONFIG_PCAP)
> #ifdef _WIN32
> #define WPCAP 1
> #endif
> #include <pcap.h>
> #endif

Thanks for the tip, again!

> I noticed that if no ifname is passed you try to get one with
> pcap_lookupdev. This is broken on winpcap, please see
> http://www.winpcap.org/pipermail/winpcap-bugs/2006-May/000220.html

I don't think it is "broken" on Windows.  The OP was just saying that 
it returns device name in wide characters on Windows.  In fact, both 
tcpdump (3.9.8) and WinDump (3.9.5) seem to do the same (trimmed 
formatting):

-----------------
	if (device == NULL) {
		device = pcap_lookupdev(ebuf);
			if (device == NULL)
				error("%s", ebuf);
	}
#ifdef WIN32
	//we assume that an ASCII string is always longer than 1 char
	if(strlen(device) == 1)
	{	//a Unicode string has a \0 as second byte (so strlen() is 1)
		fprintf(stderr, "%s: listening on %ws\n", program_name, device);
	}
	else
	{
		fprintf(stderr, "%s: listening on %s\n", program_name, device);
	}

	fflush(stderr);
#endif /* WIN32 */
	*ebuf = '\0';
	pd = pcap_open_live(device, snaplen, !pflag, 1000, ebuf);
	if (pd == NULL)
		error("%s", ebuf);
	else if (*ebuf)
		warning("%s", ebuf);
-----------------

The OP had to use %ws format, not just %s, it seems.  In fact, WinPcap 
seems to convert ASCII names to Unicode names unconditionally from 
pcap_lookupdev():

	/*
	 * Windows NT (NT 4.0, W2K, WXP). Convert the names to UNICODE for 
backward compatibility
	 */

> I am not sure if pcap_lookupdev() gets you the "right" interface on
> non-win32, so maybe just require ifname (tap does it too)?

At least, it works for me on FreeBSD. :-) Does WinDump work without 
specifying interface name when there is only one network device?  If 
it does, I am not going to change it.

> Otherwise it seems to compile and run fine.

Please try the updated patch and let me know:

http://people.freebsd.org/~jkim/qemu-pcap-20080718.diff

Note I had to re-shuffle the patch a bit to make it applicable to 
slightly older sources.

Juergen,

I simplified FreeBSD ports patch with the above patch:

http://people.freebsd.org/~jkim/qemu-devel-20080620_1-pcap.diff

Thanks!

Jung-uk Kim

[Qemu-devel] Re: [PATCH] networking using libpcap

[Jung-uk Kim]

On Friday 18 July 2008 04:39 pm, Jung-uk Kim wrote:
> On Friday 18 July 2008 12:12 pm, Sebastian Herbszt wrote:
> > Using pcap_setfilter helps and colinux (conet-bridged-daemon)
> > does use one too. Currently your filter is
> > "ether dst 52:54:00:12:34:56 or ((broadcast or multicast) and not
> > ether src 52:54:00:12:34:56)". The filter used by colinux is
> > "(ether dst 00:ff:81:24:00:00) or (ether broadcast or multicast)
> > or (ip broadcast or multicast)".
> >
> > The "and not ether src 52:54:00:12:34:56" part in your filter
> > prevents the VM from seeing own packets. It doesn't reply to own
> > "ping broadcast" where it does in colinux and VMware Server.
>
> I knew that I might have missed some edge cases. ;-P Corrected and
> enabled by default.

I simplified the filter as:

"ether dst <MAC_ADDR> or multicast"

because we are only interested in Ethernet packets and broadcast 
address is a special multicast address.  Any upper layer protocols 
must be encapsulated in one of these forms.  For example:

http://www-uxsup.csx.cam.ac.uk/courses/ipv6_basics/x84.html

Jung-uk Kim

[Qemu-devel] Re: [PATCH] networking using libpcap

[Sebastian Herbszt]

Jung-uk Kim wrote:

>> Using pcap_setfilter helps and colinux (conet-bridged-daemon) does
>> use one too. Currently your filter is
>> "ether dst 52:54:00:12:34:56 or ((broadcast or multicast) and not
>> ether src 52:54:00:12:34:56)". The filter used by colinux is
>> "(ether dst 00:ff:81:24:00:00) or (ether broadcast or multicast) or
>> (ip broadcast or multicast)".
>>
>> The "and not ether src 52:54:00:12:34:56" part in your filter
>> prevents the VM from seeing own packets. It doesn't reply to own
>> "ping broadcast" where it does in colinux and VMware Server.
> 
> I knew that I might have missed some edge cases. ;-P Corrected and 
> enabled by default.

I noticed you removed PCAP_SET_FILTER. Could people benefit from
disabling the filter? I am not sure but it might be worth to revive
PCAP_SET_FILTER and set its default value to 1. Or if there are valid
configurations which do need the filter disabled even turn (later) it into a
"filter=off" parameter.

>> I noticed that if no ifname is passed you try to get one with
>> pcap_lookupdev. This is broken on winpcap, please see
>> http://www.winpcap.org/pipermail/winpcap-bugs/2006-May/000220.html
> 
> I don't think it is "broken" on Windows.  The OP was just saying that 
> it returns device name in wide characters on Windows.

I was more concerned with "the behaviour is in any case different from
Linux because it reports all the adapters, while pcap_lookupdev on unix
reports just the first adapter" part, but did just test it and it seems to
have been fixed since.

> In fact, both 
> tcpdump (3.9.8) and WinDump (3.9.5) seem to do the same (trimmed 
> formatting):
> 
> -----------------
> if (device == NULL) {
> device = pcap_lookupdev(ebuf);
> if (device == NULL)
> error("%s", ebuf);
> }
> #ifdef WIN32
> //we assume that an ASCII string is always longer than 1 char
> if(strlen(device) == 1)
> { //a Unicode string has a \0 as second byte (so strlen() is 1)
> fprintf(stderr, "%s: listening on %ws\n", program_name, device);
> }
> else
> {
> fprintf(stderr, "%s: listening on %s\n", program_name, device);
> }
> 
> fflush(stderr);
> #endif /* WIN32 */
> *ebuf = '\0';
> pd = pcap_open_live(device, snaplen, !pflag, 1000, ebuf);
> if (pd == NULL)
> error("%s", ebuf);
> else if (*ebuf)
> warning("%s", ebuf);
> -----------------
> 
> The OP had to use %ws format, not just %s, it seems.  In fact, WinPcap 
> seems to convert ASCII names to Unicode names unconditionally from 
> pcap_lookupdev():
> 
> /*
> * Windows NT (NT 4.0, W2K, WXP). Convert the names to UNICODE for 
> backward compatibility
> */

and converts those back to ASCII. pcap-win32.c pcap_create() has

 if (strlen(device) == 1)
 {
  /*
   * It's probably a unicode string
   * Convert to ascii and pass it to pcap_create_common
   *
   * This wonderful hack is needed because pcap_lookupdev still returns
   * unicode strings, and it's used by windump when no device is specified
   * in the command line
   */

>> I am not sure if pcap_lookupdev() gets you the "right" interface on
>> non-win32, so maybe just require ifname (tap does it too)?
> 
> At least, it works for me on FreeBSD. :-) Does WinDump work without 
> specifying interface name when there is only one network device?  If 
> it does, I am not going to change it.

WinDump-3.9.5.exe: listening on \Device\NPF_GenericDialupAdapter

I have more than one adapter present, but i think the above one is present
in all Windows configurations and get's returned by pcap_lookupdev() as the
first adapter. One of the correct adapaters for me would be
\Device\NPF_{E8D10154-6C93-4BB1-808F-9816E1CB076C}.

- Sebastian

Re: [Qemu-devel] Re: [PATCH] networking using libpcap

[Anthony Liguori]

Jung-uk Kim wrote:
>> I just discovered this patch
>>
>>     
> http://lists.freebsd.org/pipermail/freebsd-emulation/2007-February/003107.html
>   
>>     that implements network access using libpcap.
>>     
>
> Since someone showed interest, I updated my patches against trunk. :-)
>
> http://people.freebsd.org/~jkim/qemu-pcap-20080717.diff
>
> I turned it off by default for now.  If you want to enable it, do:
>
> configure --enable-pcap
>   

Please have it be enabled by default and autodetect whether libpcap is 
available (like most other things in QEMU).

Also, if you'd like the patch to be applied, please send it to 
qemu-devel as an attached patch including an appropriate description and 
Signed-off-by line.

Regards,

Anthony Liguori

>> Works perfect for me and allows access to the local Ethernet right
>> out of the box, very much unlike tap and bridging. The attached
>> version applies to trunk.
>>
>> I have modified (e.g. got rid of threads) the original patch from
>> the forum and am using it here on Windows. It works fine but
>> performance is pretty low.
>>     
>
> *After* applying the new patch:
>
> cp -p vl.c vl.c.orig
> sed -e 's/#ifdef PCAP_SET_FILTER/#if 1/g' vl.c > vl.c.tmp
> mv vl.c.tmp vl.c
>
> and try again?  BTW, I have no real experience with WinPcap, so don't 
> kill me if it does not work for you. ;-)
>  
>   
>> +ifdef CONFIG_PCAP
>> +LIBS+=-lpcap
>> +endif
>>
>> On Windows it should be -lwpcap.
>>     
>
> Thanks for the tip!
>
>   
>> +    if ((fd = pcap_get_selectable_fd(s->handle)) < 0) {
>> + fprintf(stderr, "qemu: pcap_get_selectable_fd failed\n");
>> + goto fail;
>> +    }
>> +    qemu_set_fd_handler(fd, pcap_send, NULL, s);
>>
>> pcap_get_selectable_fd() is not available on Windows. I just put
>> pcap_send() in main_loop_wait().
>>     
>
> I added WinPcap API support from WinPcap manual pages but I have no 
> way of checking.  Can you try the patch and letting me know?
>
> Juergen,
>
> I made FreeBSD ports patch for emulators/qemu-devel:
>
> http://people.freebsd.org/~jkim/qemu-devel-20080620-pcap.diff
>
> FYI...
>
> Thanks,
>
> Jung-uk Kim
>
>
>   

Re: [Qemu-devel] Re: [PATCH] networking using libpcap

[Anthony Liguori]

Jung-uk Kim wrote:
> On Friday 18 July 2008 12:12 pm, Sebastian Herbszt wrote:
>
> I simplified FreeBSD ports patch with the above patch:
>
> http://people.freebsd.org/~jkim/qemu-devel-20080620_1-pcap.diff
>   

I spent some time with this patch looking to merge it.  It seems to have 
decent performance and is very easy to use.  Unfortunately, I ran into 
the following:

1) If I specify -net pcap,ifname=eth0 -net nic,model=rtl8139, I get an 
error, whereas if I specify -net nic,model=rtl8139 -net pcap,ifname=eth0 
it works.  The reason for this is that in net_pcap_init(), you search 
the vlan to discover what the IP address is of the NIC on the vlan is.

This is a big no-no as you can have one or more NICs on a single vlan.  
Moreover, NICs can be added and removed from a vlan long after it's 
created.  The patch needs a fair bit more work to handle this proper 
(presumably building a new filter rule as NICs are added and removed 
from the vlan).

2) When using ifname=eth0, traffic doesn't work from host=>guest nor 
from guest=>host.  This seems like a major short-coming to me.  Having 
another networking option that is easy to use but doesn't work in some 
fundamental way that a user would expect seems like it's just going to 
lead to even further confusion.  This patch isn't attractive to me 
unless it works just like you were bridging to a physical interface (and 
I don't think there's anything fundamental preventing that).

If you can fix these issues, I'd be happy to merge this feature.

Regards,

Anthony Liguori

> Thanks!
>
> Jung-uk Kim
>
>
>   

Re: [Qemu-devel] Re: [PATCH] networking using libpcap

[Sebastian Herbszt]

Anthony,

> 2) When using ifname=eth0, traffic doesn't work from host=>guest nor 
> from guest=>host.  This seems like a major short-coming to me.  Having 
> another networking option that is easy to use but doesn't work in some 
> fundamental way that a user would expect seems like it's just going to 
> lead to even further confusion.  This patch isn't attractive to me 
> unless it works just like you were bridging to a physical interface (and 
> I don't think there's anything fundamental preventing that).

this works on Win XP. According to the source it seems to need BIOCFEEDBACK
("This is necessary to connect host and guest.") on BSD. Maybe something like
this is also needed on your system.

Which OS did you try it with? Linux?

- Sebastian

Re: [Qemu-devel] Re: [PATCH] networking using libpcap

[Anthony Liguori]

Sebastian Herbszt wrote:
> Anthony,
>
>> 2) When using ifname=eth0, traffic doesn't work from host=>guest nor 
>> from guest=>host.  This seems like a major short-coming to me.  
>> Having another networking option that is easy to use but doesn't work 
>> in some fundamental way that a user would expect seems like it's just 
>> going to lead to even further confusion.  This patch isn't attractive 
>> to me unless it works just like you were bridging to a physical 
>> interface (and I don't think there's anything fundamental preventing 
>> that).
>
> this works on Win XP. According to the source it seems to need 
> BIOCFEEDBACK
> ("This is necessary to connect host and guest.") on BSD. Maybe 
> something like
> this is also needed on your system.
>
> Which OS did you try it with? Linux?

Yes, with libpcap 0.9.8-2.fc9.

Regards,

Anthony Liguori

> - Sebastian
>

Re: [Qemu-devel] Re: [PATCH] networking using libpcap

[Paul Brook]

> 2) When using ifname=eth0, traffic doesn't work from host=>guest nor
> from guest=>host.  This seems like a major short-coming to me.

IIRC this is a known limitation of some common libcap implementations (in 
particular Linux).

Paul