* Re: [PATCH] ethtool: Add "-f" option to flash a firmware image from the specified file to a device.
From: Ben Hutchings @ 2009-08-19 17:23 UTC (permalink / raw)
To: Ajit Khaparde; +Cc: davem, jgarzik, netdev
In-Reply-To: <20090818112522.GA22134@serverengines.com>
On Tue, 2009-08-18 at 16:55 +0530, Ajit Khaparde wrote:
> This patch adds a new "-f" option to the ethtool utility
> to flash a firmware image specified by a file, to a network device.
> The filename is passed to the network driver which will flash the image
> on the chip using the request_firmware path.
> The region to be flashed - like redboot, phy can be specified as an argument,
> which will be passed to the driver.
> More options for other flash regions can be added in future.
> The default behavior is to flash all the regions on the chip.
>
> Usage:
> ethtool -f <interface name> <filename of firmware image>
>
> ethtool -f <interface name> <filename of firmware image> [ all|redboot|phy ]
>
> Signed-off-by: Ajit Khaparde <ajitk@serverengines.com>
> ---
> ethtool-copy.h | 16 ++++++++++++
> ethtool.c | 71 +++++++++++++++++++++++++++++++++++++++++++++++++++++++-
> 2 files changed, 86 insertions(+), 1 deletions(-)
>
> diff --git a/ethtool-copy.h b/ethtool-copy.h
> index 3ca4e2c..a7d11fb 100644
> --- a/ethtool-copy.h
> +++ b/ethtool-copy.h
> @@ -272,6 +272,21 @@ struct ethtool_perm_addr {
> __u8 data[0];
> };
>
> +#define ETHTOOL_FLASH_MAX_FILENAME 128
> +#define ETHTOOL_FLASH_OP_TYPE_SIZE 32
These are missing from your patch to the in-kernel ethtool.h. What do
they mean? They are inconsistent with the ethtool_flash::data field.
[...]
> @@ -516,6 +525,10 @@ static void parse_cmdline(int argc, char **argp)
> if (phys_id_time < 0)
> show_usage(1);
> break;
> + } else if (mode == MODE_FLASHDEV) {
> + sprintf(flash_file, "%s", argp[i]);
This is missing a length check.
[...]
> @@ -2398,6 +2424,49 @@ static int do_grxclass(int fd, struct ifreq
> *ifr)
> return 0;
> }
>
> +static int do_flash(int fd, struct ifreq *ifr)
> +{
> + struct ethtool_flash efl;
> + int err;
> + char path[ETHTOOL_FLASH_MAX_FILENAME * 2] = "/lib/firmware/";
> + FILE *f;
> +
> + if (flash < 0) {
> + fprintf(stdout, "Missing filename argument\n");
> + show_usage(1);
> + return -EPERM;
This doesn't follow the ethtool convention for error codes, which is to
use unique positive numbers.
> + }
> +
> + strcat(path, flash_file);
> +
> + if (strlen(flash_file) > ETHTOOL_FLASH_MAX_FILENAME) {
> + fprintf(stdout, "Filename too long\n");
> + return -EINVAL;
> + }
> +
> + f = fopen(path, "r");
> + if (!f) {
> + perror(path);
> + return -ENOENT;
> + }
> + fclose(f);
request_firmware() relies on a user-space agent to load firmware;
normally this is udev's firmware_agent which can be configured to look
in directories other than /lib/firmware/. So this check is too strict.
The driver should report back any error from request_firmware(), so it
should not be necessary to check here at all.
> + efl.cmd = ETHTOOL_FLASHDEV;
> + sprintf(efl.data, "%s", flash_file);
[...]
What's wrong with strcpy() anyway?
Ben.
--
Ben Hutchings, Senior Software Engineer, Solarflare Communications
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.
^ permalink raw reply
* Re: [Alacrityvm-devel] [PATCH v3 3/6] vbus: add a "vbus-proxy" bus model for vbus_driver objects
From: Avi Kivity @ 2009-08-19 16:38 UTC (permalink / raw)
To: Ira W. Snyder
Cc: Michael S. Tsirkin, Gregory Haskins, kvm, netdev, linux-kernel,
alacrityvm-devel, Anthony Liguori, Ingo Molnar, Gregory Haskins,
Hollis Blanchard
In-Reply-To: <20090819162902.GB22294@ovro.caltech.edu>
On 08/19/2009 07:29 PM, Ira W. Snyder wrote:
>
>
>> virtio-$yourhardware or maybe virtio-dma
>>
>>
> How about virtio-phys?
>
Could work.
> Arnd and BenH are both looking at PPC systems (similar to mine). Grant
> Likely is looking at talking to an processor core running on an FPGA,
> IIRC. Most of the code can be shared, very little should need to be
> board-specific, I hope.
>
Excellent.
>>> That said, I'm not sure how qemu-system-ppc running on x86 could
>>> possibly communicate using virtio-net. This would mean the guest is an
>>> emulated big-endian PPC, while the host is a little-endian x86. I
>>> haven't actually tested this situation, so perhaps I am wrong.
>>>
>>>
>> I'm confused now. You don't actually have any guest, do you, so why
>> would you run qemu at all?
>>
>>
> I do not run qemu. I am just stating a problem with virtio-net that I
> noticed. This is just so someone more knowledgeable can be aware of the
> problem.
>
>
Ah, it certainly doesn't byteswap. Maybe nobody tried it. Hollis?
--
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.
^ permalink raw reply
* Re: mlx4 2.6.31-rc5: SW2HW_EQ failed.
From: Christoph Lameter @ 2009-08-19 16:29 UTC (permalink / raw)
To: Roland Dreier; +Cc: netdev, Yevgeny Petrilin
In-Reply-To: <adaeir7256h.fsf@cisco.com>
On Wed, 19 Aug 2009, Roland Dreier wrote:
> > We ran an old 2.6.26 debian kernel on this and it worked fine.
>
> The change to mlx4 to use multiple completion interrupts went in around
> 2.6.29 I think. So that sort of explains why things would work with 2.6.26.
I also see a 2 usecs latency increase with 2.6.31 vs 2.6.30.
^ permalink raw reply
* Re: [Alacrityvm-devel] [PATCH v3 3/6] vbus: add a "vbus-proxy" bus model for vbus_driver objects
From: Ira W. Snyder @ 2009-08-19 16:29 UTC (permalink / raw)
To: Avi Kivity
Cc: Michael S. Tsirkin, Gregory Haskins, kvm, netdev, linux-kernel,
alacrityvm-devel, Anthony Liguori, Ingo Molnar, Gregory Haskins
In-Reply-To: <4A8C1C22.3010101@redhat.com>
On Wed, Aug 19, 2009 at 06:37:06PM +0300, Avi Kivity wrote:
> On 08/19/2009 06:28 PM, Ira W. Snyder wrote:
>>
>>> Well, if you can't do that, you can't use virtio-pci on the host.
>>> You'll need another virtio transport (equivalent to "fake pci" you
>>> mentioned above).
>>>
>>>
>> Ok.
>>
>> Is there something similar that I can study as an example? Should I look
>> at virtio-pci?
>>
>>
>
> There's virtio-lguest, virtio-s390, and virtio-vbus.
>
>>> I think you tried to take two virtio-nets and make them talk together?
>>> That won't work. You need the code from qemu to talk to virtio-net
>>> config space, and vhost-net to pump the rings.
>>>
>>>
>> It *is* possible to make two unmodified virtio-net's talk together. I've
>> done it, and it is exactly what the virtio-over-PCI patch does. Study it
>> and you'll see how I connected the rx/tx queues together.
>>
>
> Right, crossing the cables works, but feature negotiation is screwed up,
> and both sides think the data is in their RAM.
>
> vhost-net doesn't do negotiation and doesn't assume the data lives in
> its address space.
>
Yes, that is exactly what I did: crossed the cables (in software).
I'll take a closer look at vhost-net now, and make sure I understand how
it works.
>>> Please find a name other than virtio-over-PCI since it conflicts with
>>> virtio-pci. You're tunnelling virtio config cycles (which are usually
>>> done on pci config cycles) on a new protocol which is itself tunnelled
>>> over PCI shared memory.
>>>
>>>
>> Sorry about that. Do you have suggestions for a better name?
>>
>>
>
> virtio-$yourhardware or maybe virtio-dma
>
How about virtio-phys?
Arnd and BenH are both looking at PPC systems (similar to mine). Grant
Likely is looking at talking to an processor core running on an FPGA,
IIRC. Most of the code can be shared, very little should need to be
board-specific, I hope.
>> I called it virtio-over-PCI in my previous postings to LKML, so until a
>> new patch is written and posted, I'll keep referring to it by the name
>> used in the past, so people can search for it.
>>
>> When I post virtio patches, should I CC another mailing list in addition
>> to LKML?
>>
>
> virtualization@lists.linux-foundation.org is virtio's home.
>
>> That said, I'm not sure how qemu-system-ppc running on x86 could
>> possibly communicate using virtio-net. This would mean the guest is an
>> emulated big-endian PPC, while the host is a little-endian x86. I
>> haven't actually tested this situation, so perhaps I am wrong.
>>
>
> I'm confused now. You don't actually have any guest, do you, so why
> would you run qemu at all?
>
I do not run qemu. I am just stating a problem with virtio-net that I
noticed. This is just so someone more knowledgeable can be aware of the
problem.
>>> The x86 side only needs to run virtio-net, which is present in RHEL 5.3.
>>> You'd only need to run virtio-tunnel or however it's called. All the
>>> eventfd magic takes place on the PCI agents.
>>>
>>>
>> I can upgrade the kernel to anything I want on both the x86 and ppc's.
>> I'd like to avoid changing the x86 (RHEL5) userspace, though. On the
>> ppc's, I have full control over the userspace environment.
>>
>
> You don't need any userspace on virtio-net's side.
>
> Your ppc boards emulate a virtio-net device, so all you need is the
> virtio-net module (and virtio bindings). If you chose to emulate, say,
> an e1000 card all you'd need is the e1000 driver.
>
Thanks for the replies.
Ira
^ permalink raw reply
* [PATCH v2] net: fix ks8851 build errors
From: Randy Dunlap @ 2009-08-19 15:52 UTC (permalink / raw)
To: David Miller; +Cc: sfr, netdev, akpm, linux-next, linux-kernel
In-Reply-To: <20090818.201200.69809801.davem@davemloft.net>
From: Randy Dunlap <randy.dunlap@oracle.com>
Fix build errors due to missing Kconfig select of CRC32:
ks8851.c:(.text+0x7d2ee): undefined reference to `crc32_le'
ks8851.c:(.text+0x7d2f5): undefined reference to `bitrev32'
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
---
drivers/net/Kconfig | 1 +
1 file changed, 1 insertion(+)
--- linux-next-20090818.orig/drivers/net/Kconfig
+++ linux-next-20090818/drivers/net/Kconfig
@@ -1733,6 +1733,7 @@ config KS8851
tristate "Micrel KS8851 SPI"
depends on SPI
select MII
+ select CRC32
help
SPI driver for Micrel KS8851 SPI attached network chip.
---
~Randy
LPC 2009, Sept. 23-25, Portland, Oregon
http://linuxplumbersconf.org/2009/
^ permalink raw reply
* Re: mlx4 2.6.31-rc5: SW2HW_EQ failed.
From: Christoph Lameter @ 2009-08-19 15:47 UTC (permalink / raw)
To: Roland Dreier; +Cc: netdev, Yevgeny Petrilin
In-Reply-To: <adaeir7256h.fsf@cisco.com>
On Wed, 19 Aug 2009, Roland Dreier wrote:
> > although the box does not have any slots for "hotplugging".
>
> I guess you could try booting with "possible_cpus=16" and see how that
> affects things...
I configured the kernel with a maximum of 16 cpus and things are fine.
> Sounds like the patch lets you get farther? What kind of hangs do you
> get? Still mlx4-related?
System hangs when unloading the mlx4 driver f.e.
^ permalink raw reply
* Re: mlx4 2.6.31-rc5: SW2HW_EQ failed.
From: Roland Dreier @ 2009-08-19 15:29 UTC (permalink / raw)
To: Christoph Lameter; +Cc: netdev, Yevgeny Petrilin
In-Reply-To: <alpine.DEB.1.10.0908190743410.5621@gentwo.org>
> Correct. My Dell R620 may have some weird ACPI info.
>
> SMP: Allowing 32 CPUs, 16 hotplug CPUs
>
> although the box does not have any slots for "hotplugging".
I guess you could try booting with "possible_cpus=16" and see how that
affects things...
> The patch does not fix the issue so far. We are having various hangs and
> are trying to figure out what is gong on.
Sounds like the patch lets you get farther? What kind of hangs do you
get? Still mlx4-related?
> We ran an old 2.6.26 debian kernel on this and it worked fine.
The change to mlx4 to use multiple completion interrupts went in around
2.6.29 I think. So that sort of explains why things would work with 2.6.26.
- R.
^ permalink raw reply
* Re: [Alacrityvm-devel] [PATCH v3 3/6] vbus: add a "vbus-proxy" bus model for vbus_driver objects
From: Avi Kivity @ 2009-08-19 15:37 UTC (permalink / raw)
To: Ira W. Snyder
Cc: Michael S. Tsirkin, Gregory Haskins, kvm, netdev, linux-kernel,
alacrityvm-devel, Anthony Liguori, Ingo Molnar, Gregory Haskins
In-Reply-To: <20090819152811.GA22294@ovro.caltech.edu>
On 08/19/2009 06:28 PM, Ira W. Snyder wrote:
>
>> Well, if you can't do that, you can't use virtio-pci on the host.
>> You'll need another virtio transport (equivalent to "fake pci" you
>> mentioned above).
>>
>>
> Ok.
>
> Is there something similar that I can study as an example? Should I look
> at virtio-pci?
>
>
There's virtio-lguest, virtio-s390, and virtio-vbus.
>> I think you tried to take two virtio-nets and make them talk together?
>> That won't work. You need the code from qemu to talk to virtio-net
>> config space, and vhost-net to pump the rings.
>>
>>
> It *is* possible to make two unmodified virtio-net's talk together. I've
> done it, and it is exactly what the virtio-over-PCI patch does. Study it
> and you'll see how I connected the rx/tx queues together.
>
Right, crossing the cables works, but feature negotiation is screwed up,
and both sides think the data is in their RAM.
vhost-net doesn't do negotiation and doesn't assume the data lives in
its address space.
>> Please find a name other than virtio-over-PCI since it conflicts with
>> virtio-pci. You're tunnelling virtio config cycles (which are usually
>> done on pci config cycles) on a new protocol which is itself tunnelled
>> over PCI shared memory.
>>
>>
> Sorry about that. Do you have suggestions for a better name?
>
>
virtio-$yourhardware or maybe virtio-dma
> I called it virtio-over-PCI in my previous postings to LKML, so until a
> new patch is written and posted, I'll keep referring to it by the name
> used in the past, so people can search for it.
>
> When I post virtio patches, should I CC another mailing list in addition
> to LKML?
>
virtualization@lists.linux-foundation.org is virtio's home.
> That said, I'm not sure how qemu-system-ppc running on x86 could
> possibly communicate using virtio-net. This would mean the guest is an
> emulated big-endian PPC, while the host is a little-endian x86. I
> haven't actually tested this situation, so perhaps I am wrong.
>
I'm confused now. You don't actually have any guest, do you, so why
would you run qemu at all?
>> The x86 side only needs to run virtio-net, which is present in RHEL 5.3.
>> You'd only need to run virtio-tunnel or however it's called. All the
>> eventfd magic takes place on the PCI agents.
>>
>>
> I can upgrade the kernel to anything I want on both the x86 and ppc's.
> I'd like to avoid changing the x86 (RHEL5) userspace, though. On the
> ppc's, I have full control over the userspace environment.
>
You don't need any userspace on virtio-net's side.
Your ppc boards emulate a virtio-net device, so all you need is the
virtio-net module (and virtio bindings). If you chose to emulate, say,
an e1000 card all you'd need is the e1000 driver.
--
error compiling committee.c: too many arguments to function
^ permalink raw reply
* Re: 100Mbit ethernet performance on embedded devices
From: Jamie Lokier @ 2009-08-19 15:35 UTC (permalink / raw)
To: Johannes Stezenbach; +Cc: linux-embedded, netdev
In-Reply-To: <20090819145057.GA25400@sig21.net>
Johannes Stezenbach wrote:
> a while ago I was working on a SoC with 200MHz ARM926EJ-S CPU
> and integrated 100Mbit ethernet core, connected on internal
> (fast) memory bus, with DMA. With iperf I measured:
>
> TCP RX ~70Mbit/sec (iperf -s on SoC, iperf -c on destop PC)
> TCP TX ~56Mbit/sec (iperf -s on destop PC, iperf -c o SoC)
>
> The CPU load during the iperf test is around
> 1% user, 44% system, 4% irq, 48% softirq, with 7500 irqs/sec.
>
> The kernel used in these measurements does not have iptables
> support, I think packet filtering will slow it down noticably,
> but I didn't actually try. The ethernet driver uses NAPI,
> but it doesn't seem to be a win judging from the irq/sec number.
You should see far fewer interrupts if NAPI was working properly.
Rather than NAPI not being a win, it looks like it's not active at
all.
7500/sec is close to the packet rate, for sending TCP with
full-size ethernet packages over a 100Mbit ethernet link.
> What I'm interested in are some numbers for similar hardware,
> to find out if my hardware and/or ethernet driver can be improved,
> or if the CPU will always be the limiting factor.
I have a SoC with a 166MHz ARMv4 (ARM7TDMI I think, but I'm not sure),
and an external RTL8139 100Mbit ethernet chip over the SoC's PCI bus.
It gets a little over 80Mbit/s actual data throughput in both
directions, running a simple FTP client.
> I'd also be interested to know if hardware checksumming
> support would improve throughput noticably in such a system,
> or if it is only useful for 1Gbit and above.
>
> Did anyone actually manage to get close to 100Mbit/sec
> with similar CPU resources?
Remember, the TCP throughput cannot reach 100Mbit/sec due to the
overhead of packet framing. But it should be much closer to 100 than 70.
-- Jamie
^ permalink raw reply
* Re: [Alacrityvm-devel] [PATCH v3 3/6] vbus: add a "vbus-proxy" bus model for vbus_driver objects
From: Ira W. Snyder @ 2009-08-19 15:28 UTC (permalink / raw)
To: Avi Kivity
Cc: Michael S. Tsirkin, Gregory Haskins, kvm, netdev, linux-kernel,
alacrityvm-devel, Anthony Liguori, Ingo Molnar, Gregory Haskins
In-Reply-To: <4A8B9051.3020505@redhat.com>
On Wed, Aug 19, 2009 at 08:40:33AM +0300, Avi Kivity wrote:
> On 08/19/2009 03:38 AM, Ira W. Snyder wrote:
>> On Wed, Aug 19, 2009 at 12:26:23AM +0300, Avi Kivity wrote:
>>
>>> On 08/18/2009 11:59 PM, Ira W. Snyder wrote:
>>>
>>>> On a non shared-memory system (where the guest's RAM is not just a chunk
>>>> of userspace RAM in the host system), virtio's management model seems to
>>>> fall apart. Feature negotiation doesn't work as one would expect.
>>>>
>>>>
>>> In your case, virtio-net on the main board accesses PCI config space
>>> registers to perform the feature negotiation; software on your PCI cards
>>> needs to trap these config space accesses and respond to them according
>>> to virtio ABI.
>>>
>>>
>> Is this "real PCI" (physical hardware) or "fake PCI" (software PCI
>> emulation) that you are describing?
>>
>>
>
> Real PCI.
>
>> The host (x86, PCI master) must use "real PCI" to actually configure the
>> boards, enable bus mastering, etc. Just like any other PCI device, such
>> as a network card.
>>
>> On the guests (ppc, PCI agents) I cannot add/change PCI functions (the
>> last .[0-9] in the PCI address) nor can I change PCI BAR's once the
>> board has started. I'm pretty sure that would violate the PCI spec,
>> since the PCI master would need to re-scan the bus, and re-assign
>> addresses, which is a task for the BIOS.
>>
>
> Yes. Can the boards respond to PCI config space cycles coming from the
> host, or is the config space implemented in silicon and immutable?
> (reading on, I see the answer is no). virtio-pci uses the PCI config
> space to configure the hardware.
>
Yes, the PCI config space is implemented in silicon. I can change a few
things (mostly PCI BAR attributes), but not much.
>>> (There's no real guest on your setup, right? just a kernel running on
>>> and x86 system and other kernels running on the PCI cards?)
>>>
>>>
>> Yes, the x86 (PCI master) runs Linux (booted via PXELinux). The ppc's
>> (PCI agents) also run Linux (booted via U-Boot). They are independent
>> Linux systems, with a physical PCI interconnect.
>>
>> The x86 has CONFIG_PCI=y, however the ppc's have CONFIG_PCI=n. Linux's
>> PCI stack does bad things as a PCI agent. It always assumes it is a PCI
>> master.
>>
>> It is possible for me to enable CONFIG_PCI=y on the ppc's by removing
>> the PCI bus from their list of devices provided by OpenFirmware. They
>> can not access PCI via normal methods. PCI drivers cannot work on the
>> ppc's, because Linux assumes it is a PCI master.
>>
>> To the best of my knowledge, I cannot trap configuration space accesses
>> on the PCI agents. I haven't needed that for anything I've done thus
>> far.
>>
>>
>
> Well, if you can't do that, you can't use virtio-pci on the host.
> You'll need another virtio transport (equivalent to "fake pci" you
> mentioned above).
>
Ok.
Is there something similar that I can study as an example? Should I look
at virtio-pci?
>>>> This does appear to be solved by vbus, though I haven't written a
>>>> vbus-over-PCI implementation, so I cannot be completely sure.
>>>>
>>>>
>>> Even if virtio-pci doesn't work out for some reason (though it should),
>>> you can write your own virtio transport and implement its config space
>>> however you like.
>>>
>>>
>> This is what I did with virtio-over-PCI. The way virtio-net negotiates
>> features makes this work non-intuitively.
>>
>
> I think you tried to take two virtio-nets and make them talk together?
> That won't work. You need the code from qemu to talk to virtio-net
> config space, and vhost-net to pump the rings.
>
It *is* possible to make two unmodified virtio-net's talk together. I've
done it, and it is exactly what the virtio-over-PCI patch does. Study it
and you'll see how I connected the rx/tx queues together.
The feature negotiation code also works, but in a very unintuitive
manner. I made it work in the virtio-over-PCI patch, but the devices are
hardcoded into the driver. It would be quite a bit of work to swap
virtio-net and virtio-console, for example.
>>>> I'm not at all clear on how to get feature negotiation to work on a
>>>> system like mine. From my study of lguest and kvm (see below) it looks
>>>> like userspace will need to be involved, via a miscdevice.
>>>>
>>>>
>>> I don't see why. Is the kernel on the PCI cards in full control of all
>>> accesses?
>>>
>>>
>> I'm not sure what you mean by this. Could you be more specific? This is
>> a normal, unmodified vanilla Linux kernel running on the PCI agents.
>>
>
> I meant, does board software implement the config space accesses issued
> from the host, and it seems the answer is no.
>
>
>> In my virtio-over-PCI patch, I hooked two virtio-net's together. I wrote
>> an algorithm to pair the tx/rx queues together. Since virtio-net
>> pre-fills its rx queues with buffers, I was able to use the DMA engine
>> to copy from the tx queue into the pre-allocated memory in the rx queue.
>>
>>
>
> Please find a name other than virtio-over-PCI since it conflicts with
> virtio-pci. You're tunnelling virtio config cycles (which are usually
> done on pci config cycles) on a new protocol which is itself tunnelled
> over PCI shared memory.
>
Sorry about that. Do you have suggestions for a better name?
I called it virtio-over-PCI in my previous postings to LKML, so until a
new patch is written and posted, I'll keep referring to it by the name
used in the past, so people can search for it.
When I post virtio patches, should I CC another mailing list in addition
to LKML?
>>>>
>>>>
>>> Yeah. You'll need to add byteswaps.
>>>
>>>
>> I wonder if Rusty would accept a new feature:
>> VIRTIO_F_NET_LITTLE_ENDIAN, which would allow the virtio-net driver to
>> use LE for all of it's multi-byte fields.
>>
>> I don't think the transport should have to care about the endianness.
>>
>
> Given this is not mainstream use, it would have to have zero impact when
> configured out.
>
Yes, of course.
That said, I'm not sure how qemu-system-ppc running on x86 could
possibly communicate using virtio-net. This would mean the guest is an
emulated big-endian PPC, while the host is a little-endian x86. I
haven't actually tested this situation, so perhaps I am wrong.
>> True. It's slowpath setup, so I don't care how fast it is. For reasons
>> outside my control, the x86 (PCI master) is running a RHEL5 system. This
>> means glibc-2.5, which doesn't have eventfd support, AFAIK. I could try
>> and push for an upgrade. This obviously makes cat/echo really nice, it
>> doesn't depend on glibc, only the kernel version.
>>
>> I don't give much weight to the above, because I can use the eventfd
>> syscalls directly, without glibc support. It is just more painful.
>>
>
> The x86 side only needs to run virtio-net, which is present in RHEL 5.3.
> You'd only need to run virtio-tunnel or however it's called. All the
> eventfd magic takes place on the PCI agents.
>
I can upgrade the kernel to anything I want on both the x86 and ppc's.
I'd like to avoid changing the x86 (RHEL5) userspace, though. On the
ppc's, I have full control over the userspace environment.
Thanks,
Ira
^ permalink raw reply
* Re: [PATCHv3 2/2] vhost_net: a kernel-level virtio server
From: Arnd Bergmann @ 2009-08-19 15:27 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: virtualization, netdev, kvm, linux-kernel, mingo, linux-mm, akpm,
hpa, gregory.haskins, Or Gerlitz
In-Reply-To: <20090819142038.GA3862@redhat.com>
On Wednesday 19 August 2009, Michael S. Tsirkin wrote:
> On Wed, Aug 19, 2009 at 03:46:44PM +0200, Arnd Bergmann wrote:
> > On Wednesday 19 August 2009, Michael S. Tsirkin wrote:
> >
> > Leaving that aside for now, you could replace VHOST_NET_SET_SOCKET,
> > VHOST_SET_OWNER, VHOST_RESET_OWNER
>
> SET/RESET OWNER is still needed: otherwise if you share a descriptor
> with another process, it can corrupt your memory.
How? The point of using user threads is that you only ever access the
address space of the thread that called the ioctl.
> > and your kernel thread with a new
> > VHOST_NET_SPLICE blocking ioctl that does all the transfers in the
> > context of the calling thread.
>
> For one, you'd want a thread per virtqueue.
Don't understand why. The thread can be blocked on all four ends
of the device and wake up whenever there is some work do to in
any direction. If we have data to be transferred in both ways,
we save one thread switch. It probably won't hurt much to have one
thread per direction, but I don't see the point.
> Second, an incoming traffic might arrive on another CPU, we want
> to keep it local. I guess you would also want ioctls to wake up
> the threads spuriously ...
Outbound traffic should just stay on whatever CPU was sending it
from the guest. For inbound traffic, we should only wake up
the thread on the CPU that got the data to start with.
Why would I wake up the threads spuriously? Do you mean for
stopping the transmission or something else? I guess a pthread_kill
would be enough for shutting it down.
> > This would improve the driver on various fronts:
> >
> > - no need for playing tricks with use_mm/unuse_mm
> > - possibly fewer global TLB flushes from switch_mm, which
> > may improve performance.
>
> Why would there be less flushes?
I just read up on task->active_mm handling. There probably wouldn't
be any. I got that wrong.
> > - based on that, the ability to use any kind of file
> > descriptor that can do writev/readv or sendmsg/recvmsg
> > without the nastiness you mentioned.
>
> Yes, it's an interesting approach. As I said, need to tread very
> carefully though, I don't think all issues are figured out. For example:
> what happens if we pass our own fd here? Will refcount on file ever get
> to 0 on exit? There may be others ...
right.
> > The disadvantage of course is that you need to add a user
> > thread for each guest device to make up for the workqueue
> > that you save.
>
> More importantly, you lose control of CPU locality. Simply put, a
> natural threading model in virtualization is one thread per guest vcpu.
> Asking applications to add multiple helper threads just so they can
> block forever is wrong, IMO, as userspace has no idea which CPU
> they should be on, what priority to use, etc.
But the kernel also doesn't know this, you get the same problem in
another way. If you have multiple guests running at different priorities,
the kernel will use those priorities to do the more important transfers
first, while with a global workqueue every guest gets the same priority.
You say that the natural model is to have one thread per guest
CPU, but you have a thread per host CPU instead. If the numbers
are different, you probably lose either way. It gets worse if you
try to apply NUMA policies.
> > > > to
> > > > avoid some of the implications of kernel threads like the missing
> > > > ability to handle transfer errors in user space.
> > >
> > > Are you talking about TCP here?
> > > Transfer errors are typically asynchronous - possibly eventfd
> > > as I expose for vhost net is sufficient there.
> >
> > I mean errors in general if we allow random file descriptors to be used.
> > E.g. tun_chr_aio_read could return EBADFD, EINVAL, EFAULT, ERESTARTSYS,
> > EIO, EAGAIN and possibly others. We can handle some in kernel, others
> > should never happen with vhost_net, but if something unexpected happens
> > it would be nice to just bail out to user space.
>
> And note that there might be more than one error. I guess, that's
> another problem with trying to layer on top of vfs.
Why is that different from any other system call? We just return when
we hit the first error condition.
Arnd <><
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: Kernel oops on setting sky2 interfaces down
From: Rene Mayrhofer @ 2009-08-19 15:11 UTC (permalink / raw)
To: Mike McCormack; +Cc: netdev, Richard Leitner, Stephen Hemminger
In-Reply-To: <392fb48f0908190800j508456d6h9c520e82bc7f51e0@mail.gmail.com>
[-- Attachment #1: Type: Text/Plain, Size: 808 bytes --]
Hi Mike,
On Wednesday 19 August 2009 05:00:31 pm Mike McCormack wrote:
> There's a couple of things to try:
>
> * try the latest sky2 code from net-next-2.6
> * try adding an msleep(1) after sky2_rx_stop() in sky2_down()
> * try adding a check for rx_ring and tx_ring being NULL in
> sky2_status_intr(), and disable napi while freeing the buffers in
> sky2_down()
>
> I've got an untested, ad-hoc patch against net-next-2.6 for the last
> two bits ...
I will try all these (hopefully in the next 24h), starting with the patch you
sent me in PM, then the patch attached to this email, and finally pulling
sky2.c from net-next-2.6 and applying this patch.
best regards,
Rene
--
-------------------------------------------------
Gibraltar firewall http://www.gibraltar.at/
[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]
^ permalink raw reply
* Re: 100Mbit ethernet performance on embedded devices
From: Ben Hutchings @ 2009-08-19 15:05 UTC (permalink / raw)
To: Johannes Stezenbach; +Cc: linux-embedded, netdev
In-Reply-To: <20090819145057.GA25400@sig21.net>
On Wed, 2009-08-19 at 16:50 +0200, Johannes Stezenbach wrote:
> Hi,
>
> a while ago I was working on a SoC with 200MHz ARM926EJ-S CPU
> and integrated 100Mbit ethernet core, connected on internal
> (fast) memory bus, with DMA. With iperf I measured:
>
> TCP RX ~70Mbit/sec (iperf -s on SoC, iperf -c on destop PC)
> TCP TX ~56Mbit/sec (iperf -s on destop PC, iperf -c o SoC)
>
> The CPU load during the iperf test is around
> 1% user, 44% system, 4% irq, 48% softirq, with 7500 irqs/sec.
>
> The kernel used in these measurements does not have iptables
> support, I think packet filtering will slow it down noticably,
> but I didn't actually try. The ethernet driver uses NAPI,
> but it doesn't seem to be a win judging from the irq/sec number.
> The kernel was an ancient 2.6.20.
Which driver is this? Is it possible that it does not use NAPI
correctly?
> I tried hard, but I couldn't find any performance figures for
> comparison. (All performance figures I found refer to 1Gbit
> or 10Gbit server type systems.)
>
> What I'm interested in are some numbers for similar hardware,
> to find out if my hardware and/or ethernet driver can be improved,
> or if the CPU will always be the limiting factor.
> I'd also be interested to know if hardware checksumming
> support would improve throughput noticably in such a system,
> or if it is only useful for 1Gbit and above.
I have no recent experience with this sort of system, but checksum
offload and scatter/gather DMA support should significantly reduce both
CPU and memory bus load.
Ben.
--
Ben Hutchings, Senior Software Engineer, Solarflare Communications
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.
^ permalink raw reply
* [PATCHv4 2/2] vhost_net: a kernel-level virtio server
From: Michael S. Tsirkin @ 2009-08-19 15:03 UTC (permalink / raw)
To: netdev, virtualization, kvm, linux-kernel, mingo, linux-mm, akpm
In-Reply-To: <cover.1250693417.git.mst@redhat.com>
What it is: vhost net is a character device that can be used to reduce
the number of system calls involved in virtio networking.
Existing virtio net code is used in the guest without modification.
There's similarity with vringfd, with some differences and reduced scope
- uses eventfd for signalling
- structures can be moved around in memory at any time (good for migration)
- support memory table and not just an offset (needed for kvm)
common virtio related code has been put in a separate file vhost.c and
can be made into a separate module if/when more backends appear. I used
Rusty's lguest.c as the source for developing this part : this supplied
me with witty comments I wouldn't be able to write myself.
What it is not: vhost net is not a bus, and not a generic new system
call. No assumptions are made on how guest performs hypercalls.
Userspace hypervisors are supported as well as kvm.
How it works: Basically, we connect virtio frontend (configured by
userspace) to a backend. The backend could be a network device, or a
tun-like device. In this version I only support raw socket as a backend,
which can be bound to e.g. SR IOV, or to macvlan device. Backend is
also configured by userspace, including vlan/mac etc.
Status:
This works for me, and I haven't see any crashes.
I have not run any benchmarks yet, compared to userspace, I expect to
see improved latency (as I save up to 4 system calls per packet) but not
bandwidth/CPU (as TSO and interrupt mitigation are not supported).
Features that I plan to look at in the future:
- TSO
- interrupt mitigation
- zero copy
Acked-by: Arnd Bergmann <arnd@arndb.de>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
MAINTAINERS | 10 +
arch/x86/kvm/Kconfig | 1 +
drivers/Makefile | 1 +
drivers/vhost/Kconfig | 11 +
drivers/vhost/Makefile | 2 +
drivers/vhost/net.c | 429 ++++++++++++++++++++++++++++
drivers/vhost/vhost.c | 664 ++++++++++++++++++++++++++++++++++++++++++++
drivers/vhost/vhost.h | 108 +++++++
include/linux/Kbuild | 1 +
include/linux/miscdevice.h | 1 +
include/linux/vhost.h | 100 +++++++
11 files changed, 1328 insertions(+), 0 deletions(-)
create mode 100644 drivers/vhost/Kconfig
create mode 100644 drivers/vhost/Makefile
create mode 100644 drivers/vhost/net.c
create mode 100644 drivers/vhost/vhost.c
create mode 100644 drivers/vhost/vhost.h
create mode 100644 include/linux/vhost.h
diff --git a/MAINTAINERS b/MAINTAINERS
index b1114cf..de4587f 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -5431,6 +5431,16 @@ S: Maintained
F: Documentation/filesystems/vfat.txt
F: fs/fat/
+VIRTIO HOST (VHOST)
+P: Michael S. Tsirkin
+M: mst@redhat.com
+L: kvm@vger.kernel.org
+L: virtualization@lists.osdl.org
+L: netdev@vger.kernel.org
+S: Maintained
+F: drivers/vhost/
+F: include/linux/vhost.h
+
VIA RHINE NETWORK DRIVER
M: Roger Luethi <rl@hellgate.ch>
S: Maintained
diff --git a/arch/x86/kvm/Kconfig b/arch/x86/kvm/Kconfig
index b84e571..94f44d9 100644
--- a/arch/x86/kvm/Kconfig
+++ b/arch/x86/kvm/Kconfig
@@ -64,6 +64,7 @@ config KVM_AMD
# OK, it's a little counter-intuitive to do this, but it puts it neatly under
# the virtualization menu.
+source drivers/vhost/Kconfig
source drivers/lguest/Kconfig
source drivers/virtio/Kconfig
diff --git a/drivers/Makefile b/drivers/Makefile
index bc4205d..1551ae1 100644
--- a/drivers/Makefile
+++ b/drivers/Makefile
@@ -105,6 +105,7 @@ obj-$(CONFIG_HID) += hid/
obj-$(CONFIG_PPC_PS3) += ps3/
obj-$(CONFIG_OF) += of/
obj-$(CONFIG_SSB) += ssb/
+obj-$(CONFIG_VHOST_NET) += vhost/
obj-$(CONFIG_VIRTIO) += virtio/
obj-$(CONFIG_VLYNQ) += vlynq/
obj-$(CONFIG_STAGING) += staging/
diff --git a/drivers/vhost/Kconfig b/drivers/vhost/Kconfig
new file mode 100644
index 0000000..d955406
--- /dev/null
+++ b/drivers/vhost/Kconfig
@@ -0,0 +1,11 @@
+config VHOST_NET
+ tristate "Host kernel accelerator for virtio net"
+ depends on NET && EVENTFD
+ ---help---
+ This kernel module can be loaded in host kernel to accelerate
+ guest networking with virtio_net. Not to be confused with virtio_net
+ module itself which needs to be loaded in guest kernel.
+
+ To compile this driver as a module, choose M here: the module will
+ be called vhost_net.
+
diff --git a/drivers/vhost/Makefile b/drivers/vhost/Makefile
new file mode 100644
index 0000000..72dd020
--- /dev/null
+++ b/drivers/vhost/Makefile
@@ -0,0 +1,2 @@
+obj-$(CONFIG_VHOST_NET) += vhost_net.o
+vhost_net-y := vhost.o net.o
diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c
new file mode 100644
index 0000000..64d0c13
--- /dev/null
+++ b/drivers/vhost/net.c
@@ -0,0 +1,429 @@
+/* Copyright (C) 2009 Red Hat, Inc.
+ * Author: Michael S. Tsirkin <mst@redhat.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.
+ *
+ * virtio-net server in host kernel.
+ */
+
+#include <linux/compat.h>
+#include <linux/eventfd.h>
+#include <linux/vhost.h>
+#include <linux/virtio_net.h>
+#include <linux/mmu_context.h>
+#include <linux/miscdevice.h>
+#include <linux/module.h>
+#include <linux/mutex.h>
+#include <linux/workqueue.h>
+#include <linux/rcupdate.h>
+#include <linux/file.h>
+
+#include <linux/net.h>
+#include <linux/if_packet.h>
+#include <linux/if_arp.h>
+
+#include <net/sock.h>
+
+#include "vhost.h"
+
+enum {
+ VHOST_NET_VQ_RX = 0,
+ VHOST_NET_VQ_TX = 1,
+ VHOST_NET_VQ_MAX = 2,
+};
+
+struct vhost_net {
+ struct vhost_dev dev;
+ struct vhost_virtqueue vqs[VHOST_NET_VQ_MAX];
+ /* We use a kind of RCU to access sock pointer.
+ * All readers access it from workqueue, which makes it possible to
+ * flush the workqueue instead of synchronize_rcu. Therefore readers do
+ * not need to call rcu_read_lock/rcu_read_unlock: the beginning of
+ * work item execution acts instead of rcu_read_lock() and the end of
+ * work item execution acts instead of rcu_read_lock().
+ * Writers use device mutex. */
+ struct socket *sock;
+ struct vhost_poll poll[VHOST_NET_VQ_MAX];
+};
+
+/* Expects to be always run from workqueue - which acts as
+ * read-size critical section for our kind of RCU. */
+static void handle_tx(struct vhost_net *net)
+{
+ struct vhost_virtqueue *vq = &net->dev.vqs[VHOST_NET_VQ_TX];
+ unsigned head, out, in;
+ struct msghdr msg = {
+ .msg_name = NULL,
+ .msg_namelen = 0,
+ .msg_control = NULL,
+ .msg_controllen = 0,
+ .msg_iov = (struct iovec *)vq->iov + 1,
+ .msg_flags = MSG_DONTWAIT,
+ };
+ size_t len;
+ int err;
+ struct socket *sock = rcu_dereference(net->sock);
+ if (!sock || !sock_writeable(sock->sk))
+ return;
+
+ use_mm(net->dev.mm);
+ mutex_lock(&vq->mutex);
+ for (;;) {
+ head = vhost_get_vq_desc(&net->dev, vq, vq->iov, &out, &in);
+ /* Nothing new? Wait for eventfd to tell us they refilled. */
+ if (head == vq->num)
+ break;
+ if (out <= 1 || in) {
+ vq_err(vq, "Unexpected descriptor format for TX: "
+ "out %d, int %d\n", out, in);
+ break;
+ }
+ /* Sanity check */
+ if (vq->iov->iov_len != sizeof(struct virtio_net_hdr)) {
+ vq_err(vq, "Unexpected header len for TX: "
+ "%ld expected %zd\n", vq->iov->iov_len,
+ sizeof(struct virtio_net_hdr));
+ break;
+ }
+ /* Skip header. TODO: support TSO. */
+ msg.msg_iovlen = out - 1;
+ len = iov_length(vq->iov + 1, out - 1);
+ /* TODO: Check specific error and bomb out unless ENOBUFS? */
+ err = sock->ops->sendmsg(NULL, sock, &msg, len);
+ if (err < 0) {
+ vhost_discard_vq_desc(vq);
+ break;
+ }
+ if (err != len)
+ pr_err("Truncated TX packet: "
+ " len %d != %zd\n", err, len);
+ vhost_add_used_and_trigger(vq, head,
+ len + sizeof(struct virtio_net_hdr));
+ }
+
+ mutex_unlock(&vq->mutex);
+ unuse_mm(net->dev.mm);
+}
+
+/* Expects to be always run from workqueue - which acts as
+ * read-size critical section for our kind of RCU. */
+static void handle_rx(struct vhost_net *net)
+{
+ struct vhost_virtqueue *vq = &net->dev.vqs[VHOST_NET_VQ_RX];
+ unsigned head, out, in;
+ struct msghdr msg = {
+ .msg_name = NULL,
+ .msg_namelen = 0,
+ .msg_control = NULL, /* FIXME: get and handle RX aux data. */
+ .msg_controllen = 0,
+ .msg_iov = vq->iov + 1,
+ .msg_flags = MSG_DONTWAIT,
+ };
+
+ struct virtio_net_hdr hdr = {
+ .flags = 0,
+ .gso_type = VIRTIO_NET_HDR_GSO_NONE
+ };
+
+ size_t len;
+ int err;
+ struct socket *sock = rcu_dereference(net->sock);
+ if (!sock || skb_queue_empty(&sock->sk->sk_receive_queue))
+ return;
+
+ use_mm(net->dev.mm);
+ mutex_lock(&vq->mutex);
+
+ for (;;) {
+ head = vhost_get_vq_desc(&net->dev, vq, vq->iov, &out, &in);
+ if (head == vq->num)
+ break;
+ if (in <= 1 || out) {
+ vq_err(vq, "Unexpected descriptor format for RX: "
+ "out %d, int %d\n",
+ out, in);
+ break;
+ }
+ /* Sanity check */
+ if (vq->iov->iov_len != sizeof(struct virtio_net_hdr)) {
+ vq_err(vq, "Unexpected header len for RX: "
+ "%ld expected %zd\n",
+ vq->iov->iov_len, sizeof(struct virtio_net_hdr));
+ break;
+ }
+ /* Skip header. TODO: support TSO/mergeable rx buffers. */
+ msg.msg_iovlen = in - 1;
+ len = iov_length(vq->iov + 1, in - 1);
+ err = sock->ops->recvmsg(NULL, sock, &msg,
+ len, MSG_DONTWAIT | MSG_TRUNC);
+ /* TODO: Check specific error and bomb out unless EAGAIN? */
+ if (err < 0) {
+ vhost_discard_vq_desc(vq);
+ break;
+ }
+ /* TODO: Should check and handle checksum. */
+ if (err > len) {
+ pr_err("Discarded truncated rx packet: "
+ " len %d > %zd\n", err, len);
+ vhost_discard_vq_desc(vq);
+ continue;
+ }
+ len = err;
+ err = copy_to_user(vq->iov->iov_base, &hdr, sizeof hdr);
+ if (err) {
+ vq_err(vq, "Unable to write vnet_hdr at addr %p: %d\n",
+ vq->iov->iov_base, err);
+ break;
+ }
+ vhost_add_used_and_trigger(vq, head, len + sizeof hdr);
+ }
+
+ mutex_unlock(&vq->mutex);
+ unuse_mm(net->dev.mm);
+}
+
+static void handle_tx_kick(struct work_struct *work)
+{
+ struct vhost_virtqueue *vq;
+ struct vhost_net *net;
+ vq = container_of(work, struct vhost_virtqueue, poll.work);
+ net = container_of(vq->dev, struct vhost_net, dev);
+ handle_tx(net);
+}
+
+static void handle_rx_kick(struct work_struct *work)
+{
+ struct vhost_virtqueue *vq;
+ struct vhost_net *net;
+ vq = container_of(work, struct vhost_virtqueue, poll.work);
+ net = container_of(vq->dev, struct vhost_net, dev);
+ handle_rx(net);
+}
+
+static void handle_tx_net(struct work_struct *work)
+{
+ struct vhost_net *net;
+ net = container_of(work, struct vhost_net, poll[VHOST_NET_VQ_TX].work);
+ handle_tx(net);
+}
+
+static void handle_rx_net(struct work_struct *work)
+{
+ struct vhost_net *net;
+ net = container_of(work, struct vhost_net, poll[VHOST_NET_VQ_RX].work);
+ handle_rx(net);
+}
+
+static int vhost_net_open(struct inode *inode, struct file *f)
+{
+ struct vhost_net *n = kzalloc(sizeof *n, GFP_KERNEL);
+ int r;
+ if (!n)
+ return -ENOMEM;
+ f->private_data = n;
+ n->vqs[VHOST_NET_VQ_TX].handle_kick = handle_tx_kick;
+ n->vqs[VHOST_NET_VQ_RX].handle_kick = handle_rx_kick;
+ r = vhost_dev_init(&n->dev, n->vqs, VHOST_NET_VQ_MAX);
+ if (r < 0) {
+ kfree(n);
+ return r;
+ }
+
+ vhost_poll_init(n->poll + VHOST_NET_VQ_TX, handle_tx_net, POLLOUT);
+ vhost_poll_init(n->poll + VHOST_NET_VQ_RX, handle_rx_net, POLLIN);
+ return 0;
+}
+
+static struct socket *vhost_net_stop(struct vhost_net *n)
+{
+ struct socket *sock = n->sock;
+ rcu_assign_pointer(n->sock, NULL);
+ if (sock) {
+ vhost_poll_flush(n->poll + VHOST_NET_VQ_TX);
+ vhost_poll_flush(n->poll + VHOST_NET_VQ_RX);
+ }
+ return sock;
+}
+
+static int vhost_net_release(struct inode *inode, struct file *f)
+{
+ struct vhost_net *n = f->private_data;
+ struct socket *sock;
+
+ sock = vhost_net_stop(n);
+ vhost_dev_cleanup(&n->dev);
+ if (sock)
+ fput(sock->file);
+ kfree(n);
+ return 0;
+}
+
+static long vhost_net_set_socket(struct vhost_net *n, int fd)
+{
+ struct {
+ struct sockaddr_ll sa;
+ char buf[MAX_ADDR_LEN];
+ } uaddr;
+ struct socket *sock, *oldsock = NULL;
+ int uaddr_len = sizeof uaddr, r;
+
+ mutex_lock(&n->dev.mutex);
+ r = vhost_dev_check_owner(&n->dev);
+ if (r)
+ goto done;
+
+ if (fd == -1) {
+ /* Disconnect from socket and device. */
+ oldsock = vhost_net_stop(n);
+ goto done;
+ }
+
+ sock = sockfd_lookup(fd, &r);
+ if (!sock) {
+ r = -ENOTSOCK;
+ goto done;
+ }
+
+ /* Parameter checking */
+ if (sock->sk->sk_type != SOCK_RAW) {
+ r = -ESOCKTNOSUPPORT;
+ goto done;
+ }
+
+ r = sock->ops->getname(sock, (struct sockaddr *)&uaddr.sa,
+ &uaddr_len, 0);
+ if (r)
+ goto done;
+
+ if (uaddr.sa.sll_family != AF_PACKET) {
+ r = -EPFNOSUPPORT;
+ goto done;
+ }
+
+ /* start polling new socket */
+ if (sock == oldsock)
+ goto done;
+
+ if (oldsock) {
+ vhost_poll_stop(n->poll + VHOST_NET_VQ_TX);
+ vhost_poll_stop(n->poll + VHOST_NET_VQ_RX);
+ }
+ oldsock = n->sock;
+ rcu_assign_pointer(n->sock, sock);
+ vhost_poll_start(n->poll + VHOST_NET_VQ_TX, sock->file);
+ vhost_poll_start(n->poll + VHOST_NET_VQ_RX, sock->file);
+done:
+ mutex_unlock(&n->dev.mutex);
+ if (oldsock) {
+ vhost_poll_flush(n->poll + VHOST_NET_VQ_TX);
+ vhost_poll_flush(n->poll + VHOST_NET_VQ_RX);
+ vhost_poll_flush(&n->dev.vqs[VHOST_NET_VQ_TX].poll);
+ vhost_poll_flush(&n->dev.vqs[VHOST_NET_VQ_RX].poll);
+ fput(oldsock->file);
+ }
+ return r;
+}
+
+static long vhost_net_reset_owner(struct vhost_net *n)
+{
+ struct socket *sock = NULL;
+ long r;
+ mutex_lock(&n->dev.mutex);
+ r = vhost_dev_check_owner(&n->dev);
+ if (r)
+ goto done;
+ sock = vhost_net_stop(n);
+ r = vhost_dev_reset_owner(&n->dev);
+done:
+ mutex_unlock(&n->dev.mutex);
+ if (sock)
+ fput(sock->file);
+ return r;
+}
+
+static long vhost_net_ioctl(struct file *f, unsigned int ioctl,
+ unsigned long arg)
+{
+ struct vhost_net *n = f->private_data;
+ void __user *argp = (void __user *)arg;
+ u32 __user *featurep = argp;
+ int __user *fdp = argp;
+ u32 features;
+ int fd, r;
+ switch (ioctl) {
+ case VHOST_NET_SET_SOCKET:
+ r = get_user(fd, fdp);
+ if (r < 0)
+ return r;
+ return vhost_net_set_socket(n, fd);
+ case VHOST_GET_FEATURES:
+ /* No features for now */
+ features = 0;
+ return put_user(features, featurep);
+ case VHOST_ACK_FEATURES:
+ r = get_user(features, featurep);
+ /* No features for now */
+ if (r < 0)
+ return r;
+ if (features)
+ return -EOPNOTSUPP;
+ return 0;
+ case VHOST_RESET_OWNER:
+ return vhost_net_reset_owner(n);
+ default:
+ return vhost_dev_ioctl(&n->dev, ioctl, arg);
+ }
+}
+
+#ifdef CONFIG_COMPAT
+static long vhost_net_compat_ioctl(struct file *f, unsigned int ioctl,
+ unsigned long arg)
+{
+ return vhost_net_ioctl(f, ioctl, (unsigned long)compat_ptr(arg));
+}
+#endif
+
+const static struct file_operations vhost_net_fops = {
+ .owner = THIS_MODULE,
+ .release = vhost_net_release,
+ .unlocked_ioctl = vhost_net_ioctl,
+#ifdef CONFIG_COMPAT
+ .compat_ioctl = vhost_net_compat_ioctl,
+#endif
+ .open = vhost_net_open,
+};
+
+static struct miscdevice vhost_net_misc = {
+ VHOST_NET_MINOR,
+ "vhost-net",
+ &vhost_net_fops,
+};
+
+int vhost_net_init(void)
+{
+ int r = vhost_init();
+ if (r)
+ goto err_init;
+ r = misc_register(&vhost_net_misc);
+ if (r)
+ goto err_reg;
+ return 0;
+err_reg:
+ vhost_cleanup();
+err_init:
+ return r;
+
+}
+module_init(vhost_net_init);
+
+void vhost_net_exit(void)
+{
+ misc_deregister(&vhost_net_misc);
+ vhost_cleanup();
+}
+module_exit(vhost_net_exit);
+
+MODULE_VERSION("0.0.1");
+MODULE_LICENSE("GPL v2");
+MODULE_AUTHOR("Michael S. Tsirkin");
+MODULE_DESCRIPTION("Host kernel accelerator for virtio net");
diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
new file mode 100644
index 0000000..e14169f
--- /dev/null
+++ b/drivers/vhost/vhost.c
@@ -0,0 +1,664 @@
+/* Copyright (C) 2009 Red Hat, Inc.
+ * Copyright (C) 2006 Rusty Russell IBM Corporation
+ *
+ * Author: Michael S. Tsirkin <mst@redhat.com>
+ *
+ * Inspiration, some code, and most witty comments come from
+ * Documentation/lguest/lguest.c, by Rusty Russell
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.
+ *
+ * Generic code for virtio server in host kernel.
+ */
+
+#include <linux/eventfd.h>
+#include <linux/vhost.h>
+#include <linux/virtio_net.h>
+#include <linux/mm.h>
+#include <linux/miscdevice.h>
+#include <linux/mutex.h>
+#include <linux/workqueue.h>
+#include <linux/rcupdate.h>
+#include <linux/poll.h>
+#include <linux/file.h>
+
+#include <linux/net.h>
+#include <linux/if_packet.h>
+#include <linux/if_arp.h>
+
+#include <net/sock.h>
+
+#include "vhost.h"
+
+enum {
+ VHOST_MEMORY_MAX_NREGIONS = 64,
+};
+
+static struct workqueue_struct *vhost_workqueue;
+
+static void vhost_poll_func(struct file *file, wait_queue_head_t *wqh,
+ poll_table *pt)
+{
+ struct vhost_poll *poll;
+ poll = container_of(pt, struct vhost_poll, table);
+
+ poll->wqh = wqh;
+ add_wait_queue(wqh, &poll->wait);
+}
+
+static int vhost_poll_wakeup(wait_queue_t *wait, unsigned mode, int sync,
+ void *key)
+{
+ struct vhost_poll *poll;
+ poll = container_of(wait, struct vhost_poll, wait);
+ if (!((unsigned long)key & poll->mask))
+ return 0;
+
+ queue_work(vhost_workqueue, &poll->work);
+ return 0;
+}
+
+/* Init poll structure */
+void vhost_poll_init(struct vhost_poll *poll, work_func_t func,
+ unsigned long mask)
+{
+ INIT_WORK(&poll->work, func);
+ init_waitqueue_func_entry(&poll->wait, vhost_poll_wakeup);
+ init_poll_funcptr(&poll->table, vhost_poll_func);
+ poll->mask = mask;
+}
+
+/* Start polling a file. We add ourselves to file's wait queue. The user must
+ * keep a reference to a file until after vhost_poll_stop is called. */
+void vhost_poll_start(struct vhost_poll *poll, struct file *file)
+{
+ unsigned long mask;
+ mask = file->f_op->poll(file, &poll->table);
+ if (mask)
+ vhost_poll_wakeup(&poll->wait, 0, 0, (void *)mask);
+}
+
+/* Stop polling a file. After this function returns, it becomes safe to drop the
+ * file reference. You must also flush afterwards. */
+void vhost_poll_stop(struct vhost_poll *poll)
+{
+ remove_wait_queue(poll->wqh, &poll->wait);
+}
+
+/* Flush any work that has been scheduled. When calling this, don't hold any
+ * locks that are also used by the callback. */
+void vhost_poll_flush(struct vhost_poll *poll)
+{
+ flush_work(&poll->work);
+}
+
+long vhost_dev_init(struct vhost_dev *dev,
+ struct vhost_virtqueue *vqs, int nvqs)
+{
+ int i;
+ dev->vqs = vqs;
+ dev->nvqs = nvqs;
+ mutex_init(&dev->mutex);
+
+ for (i = 0; i < dev->nvqs; ++i) {
+ dev->vqs[i].dev = dev;
+ mutex_init(&dev->vqs[i].mutex);
+ if (dev->vqs[i].handle_kick)
+ vhost_poll_init(&dev->vqs[i].poll,
+ dev->vqs[i].handle_kick,
+ POLLIN);
+ }
+ return 0;
+}
+
+/* User should have device mutex */
+long vhost_dev_check_owner(struct vhost_dev *dev)
+{
+ return dev->mm == current->mm ? 0 : -EPERM;
+}
+
+/* User should have device mutex */
+static long vhost_dev_set_owner(struct vhost_dev *dev)
+{
+ if (dev->mm)
+ return -EBUSY;
+ dev->mm = get_task_mm(current);
+ return 0;
+}
+
+/* User should have device mutex */
+long vhost_dev_reset_owner(struct vhost_dev *dev)
+{
+ struct vhost_memory *memory;
+
+ /* Restore memory to default 1:1 mapping. */
+ memory = kmalloc(offsetof(struct vhost_memory, regions) +
+ 2 * sizeof *memory->regions, GFP_KERNEL);
+ if (!memory)
+ return -ENOMEM;
+
+ vhost_dev_cleanup(dev);
+
+ memory->nregions = 2;
+ memory->regions[0].guest_phys_addr = 1;
+ memory->regions[0].userspace_addr = 1;
+ memory->regions[0].memory_size = ~0ULL;
+ memory->regions[1].guest_phys_addr = 0;
+ memory->regions[1].userspace_addr = 0;
+ memory->regions[1].memory_size = 1;
+ dev->memory = memory;
+ return 0;
+}
+
+/* User should have device mutex */
+void vhost_dev_cleanup(struct vhost_dev *dev)
+{
+ int i;
+ for (i = 0; i < dev->nvqs; ++i) {
+ if (dev->vqs[i].kick && dev->vqs[i].handle_kick) {
+ vhost_poll_stop(&dev->vqs[i].poll);
+ vhost_poll_flush(&dev->vqs[i].poll);
+ }
+ if (dev->vqs[i].error_ctx)
+ eventfd_ctx_put(dev->vqs[i].error_ctx);
+ if (dev->vqs[i].error)
+ fput(dev->vqs[i].error);
+ if (dev->vqs[i].kick)
+ fput(dev->vqs[i].kick);
+ if (dev->vqs[i].call_ctx)
+ eventfd_ctx_put(dev->vqs[i].call_ctx);
+ if (dev->vqs[i].call)
+ fput(dev->vqs[i].call);
+ dev->vqs[i].error_ctx = NULL;
+ dev->vqs[i].error = NULL;
+ dev->vqs[i].kick = NULL;
+ dev->vqs[i].call_ctx = NULL;
+ dev->vqs[i].call = NULL;
+ }
+ /* No one will access memory at this point */
+ kfree(dev->memory);
+ dev->memory = NULL;
+ if (dev->mm)
+ mmput(dev->mm);
+ dev->mm = NULL;
+}
+
+static long vhost_set_memory(struct vhost_dev *d, struct vhost_memory __user *m)
+{
+ struct vhost_memory mem, *newmem, *oldmem;
+ unsigned long size = offsetof(struct vhost_memory, regions);
+ long r;
+ r = copy_from_user(&mem, m, size);
+ if (r)
+ return r;
+ if (mem.padding)
+ return -EOPNOTSUPP;
+ if (mem.nregions > VHOST_MEMORY_MAX_NREGIONS)
+ return -E2BIG;
+ newmem = kmalloc(size + mem.nregions * sizeof *m->regions, GFP_KERNEL);
+ if (!newmem)
+ return -ENOMEM;
+
+ memcpy(newmem, &mem, size);
+ r = copy_from_user(newmem->regions, m->regions,
+ mem.nregions * sizeof *m->regions);
+ if (r) {
+ kfree(newmem);
+ return r;
+ }
+ oldmem = d->memory;
+ rcu_assign_pointer(d->memory, newmem);
+ synchronize_rcu();
+ kfree(oldmem);
+ return 0;
+}
+
+static int init_used(struct vhost_virtqueue *vq)
+{
+ u16 flags = 0;
+ int r = put_user(flags, &vq->used->flags);
+ if (r)
+ return r;
+ return get_user(vq->last_used_idx, &vq->used->idx);
+}
+
+static long vhost_set_vring(struct vhost_dev *d, int ioctl, void __user *argp)
+{
+ struct file *eventfp, *filep = NULL,
+ *pollstart = NULL, *pollstop = NULL;
+ struct eventfd_ctx *ctx = NULL;
+ u32 __user *idxp = argp;
+ struct vhost_virtqueue *vq;
+ struct vhost_vring_state s;
+ struct vhost_vring_file f;
+ struct vhost_vring_addr a;
+ u32 idx;
+ long r;
+
+ r = get_user(idx, idxp);
+ if (r < 0)
+ return r;
+ if (idx > d->nvqs)
+ return -ENOBUFS;
+
+ vq = d->vqs + idx;
+
+ mutex_lock(&vq->mutex);
+
+ switch (ioctl) {
+ case VHOST_SET_VRING_NUM:
+ r = copy_from_user(&s, argp, sizeof s);
+ if (r < 0)
+ break;
+ if (s.num > 0xffff) {
+ r = -EINVAL;
+ break;
+ }
+ vq->num = s.num;
+ break;
+ case VHOST_SET_VRING_BASE:
+ r = copy_from_user(&s, argp, sizeof s);
+ if (r < 0)
+ break;
+ if (s.num > 0xffff) {
+ r = -EINVAL;
+ break;
+ }
+ vq->last_avail_idx = s.num;
+ break;
+ case VHOST_GET_VRING_BASE:
+ s.index = idx;
+ s.num = vq->last_avail_idx;
+ r = copy_to_user(argp, &s, sizeof s);
+ break;
+ case VHOST_SET_VRING_DESC:
+ r = copy_from_user(&a, argp, sizeof a);
+ if (r < 0)
+ break;
+ if (a.padding) {
+ r = -EOPNOTSUPP;
+ break;
+ }
+ if ((u64)(long)a.user_addr != a.user_addr) {
+ r = -EFAULT;
+ break;
+ }
+ vq->desc = (void __user *)(long)a.user_addr;
+ break;
+ case VHOST_SET_VRING_AVAIL:
+ r = copy_from_user(&a, argp, sizeof a);
+ if (r < 0)
+ break;
+ if (a.padding) {
+ r = -EOPNOTSUPP;
+ break;
+ }
+ if ((u64)(long)a.user_addr != a.user_addr) {
+ r = -EFAULT;
+ break;
+ }
+ vq->avail = (void __user *)(long)a.user_addr;
+ break;
+ case VHOST_SET_VRING_USED:
+ r = copy_from_user(&a, argp, sizeof a);
+ if (r < 0)
+ break;
+ if (a.padding) {
+ r = -EOPNOTSUPP;
+ break;
+ }
+ if ((u64)(long)a.user_addr != a.user_addr) {
+ r = -EFAULT;
+ break;
+ }
+ vq->used = (void __user *)(long)a.user_addr;
+ r = init_used(vq);
+ if (r)
+ break;
+ break;
+ case VHOST_SET_VRING_KICK:
+ r = copy_from_user(&f, argp, sizeof f);
+ if (r < 0)
+ break;
+ eventfp = f.fd == -1 ? NULL : eventfd_fget(f.fd);
+ if (IS_ERR(eventfp))
+ return PTR_ERR(eventfp);
+ if (eventfp != vq->kick) {
+ pollstop = filep = vq->kick;
+ pollstart = vq->kick = eventfp;
+ } else
+ filep = eventfp;
+ break;
+ case VHOST_SET_VRING_CALL:
+ r = copy_from_user(&f, argp, sizeof f);
+ if (r < 0)
+ break;
+ eventfp = f.fd == -1 ? NULL : eventfd_fget(f.fd);
+ if (IS_ERR(eventfp))
+ return PTR_ERR(eventfp);
+ if (eventfp != vq->call) {
+ filep = vq->call;
+ ctx = vq->call_ctx;
+ vq->call = eventfp;
+ vq->call_ctx = eventfp ?
+ eventfd_ctx_fileget(eventfp) : NULL;
+ } else
+ filep = eventfp;
+ break;
+ case VHOST_SET_VRING_ERR:
+ r = copy_from_user(&f, argp, sizeof f);
+ if (r < 0)
+ break;
+ eventfp = f.fd == -1 ? NULL : eventfd_fget(f.fd);
+ if (IS_ERR(eventfp))
+ return PTR_ERR(eventfp);
+ if (eventfp != vq->error) {
+ filep = vq->error;
+ vq->error = eventfp;
+ ctx = vq->error_ctx;
+ vq->error_ctx = eventfp ?
+ eventfd_ctx_fileget(eventfp) : NULL;
+ } else
+ filep = eventfp;
+ break;
+ default:
+ r = -ENOIOCTLCMD;
+ }
+
+ if (pollstop && vq->handle_kick)
+ vhost_poll_stop(&vq->poll);
+
+ if (ctx)
+ eventfd_ctx_put(ctx);
+ if (filep)
+ fput(filep);
+
+ if (pollstart && vq->handle_kick)
+ vhost_poll_start(&vq->poll, vq->kick);
+
+ mutex_unlock(&vq->mutex);
+
+ if (pollstop && vq->handle_kick)
+ vhost_poll_flush(&vq->poll);
+ return 0;
+}
+
+long vhost_dev_ioctl(struct vhost_dev *d, unsigned int ioctl, unsigned long arg)
+{
+ void __user *argp = (void __user *)arg;
+ long r;
+
+ mutex_lock(&d->mutex);
+ if (ioctl == VHOST_SET_OWNER) {
+ r = vhost_dev_set_owner(d);
+ goto done;
+ }
+
+ r = vhost_dev_check_owner(d);
+ if (r)
+ goto done;
+
+ switch (ioctl) {
+ case VHOST_SET_MEM_TABLE:
+ r = vhost_set_memory(d, argp);
+ break;
+ default:
+ r = vhost_set_vring(d, ioctl, argp);
+ break;
+ }
+done:
+ mutex_unlock(&d->mutex);
+ return r;
+}
+
+static const struct vhost_memory_region *find_region(struct vhost_memory *mem,
+ __u64 addr, __u32 len)
+{
+ struct vhost_memory_region *reg;
+ int i;
+ /* linear search is not brilliant, but we really have on the order of 6
+ * regions in practice */
+ for (i = 0; i < mem->nregions; ++i) {
+ reg = mem->regions + i;
+ if (reg->guest_phys_addr <= addr &&
+ reg->guest_phys_addr + reg->memory_size - 1 >= addr)
+ return reg;
+ }
+ return NULL;
+}
+
+/* FIXME: this does not handle a region that spans multiple
+ * address/len pairs */
+int translate_desc(struct vhost_dev *dev, u64 addr, u32 len,
+ struct iovec iov[], int iov_count, int iov_size,
+ unsigned *num)
+{
+ const struct vhost_memory_region *reg;
+ struct vhost_memory *mem;
+ struct iovec *_iov;
+ u64 s = 0;
+ int ret = 0;
+
+ rcu_read_lock();
+
+ mem = rcu_dereference(dev->memory);
+ while ((u64)len > s) {
+ u64 size;
+ if (*num + iov_count >= iov_size) {
+ ret = -ENOBUFS;
+ break;
+ }
+ reg = find_region(mem, addr, len);
+ if (!reg) {
+ ret = -EFAULT;
+ break;
+ }
+ _iov = iov + iov_count + *num;
+ size = reg->memory_size - addr + reg->guest_phys_addr;
+ _iov->iov_len = min((u64)len, size);
+ _iov->iov_base = (void *)
+ (reg->userspace_addr + addr - reg->guest_phys_addr);
+ s += size;
+ addr += size;
+ ++*num;
+ }
+
+ rcu_read_unlock();
+ return ret;
+}
+
+/* Each buffer in the virtqueues is actually a chain of descriptors. This
+ * function returns the next descriptor in the chain, or vq->vring.num if we're
+ * at the end. */
+static unsigned next_desc(struct vhost_virtqueue *vq, struct vring_desc *desc)
+{
+ unsigned int next;
+
+ /* If this descriptor says it doesn't chain, we're done. */
+ if (!(desc->flags & VRING_DESC_F_NEXT))
+ return vq->num;
+
+ /* Check they're not leading us off end of descriptors. */
+ next = desc->next;
+ /* Make sure compiler knows to grab that: we don't want it changing! */
+ /* We will use the result as an index in an array, so most
+ * architectures only need a compiler barrier here. */
+ read_barrier_depends();
+
+ if (next >= vq->num) {
+ vq_err(vq, "Desc next is %u > %u", next, vq->num);
+ return vq->num;
+ }
+
+ return next;
+}
+
+/* This looks in the virtqueue and for the first available buffer, and converts
+ * it to an iovec for convenient access. Since descriptors consist of some
+ * number of output then some number of input descriptors, it's actually two
+ * iovecs, but we pack them into one and note how many of each there were.
+ *
+ * This function returns the descriptor number found, or vq->num (which
+ * is never a valid descriptor number) if none was found. */
+unsigned vhost_get_vq_desc(struct vhost_dev *dev, struct vhost_virtqueue *vq,
+ struct iovec iov[],
+ unsigned int *out_num, unsigned int *in_num)
+{
+ struct vring_desc desc;
+ unsigned int i, head;
+ u16 last_avail_idx, idx;
+
+ /* Check it isn't doing very strange things with descriptor numbers. */
+ last_avail_idx = vq->last_avail_idx;
+ if (get_user(idx, &vq->avail->idx)) {
+ vq_err(vq, "Failed to access avail idx at %p\n",
+ &vq->avail->idx);
+ return vq->num;
+ }
+
+ if ((u16)(idx - last_avail_idx) > vq->num) {
+ vq_err(vq, "Guest moved used index from %u to %u",
+ last_avail_idx, idx);
+ return vq->num;
+ }
+
+ /* If there's nothing new since last we looked, return invalid. */
+ if (idx == last_avail_idx)
+ return vq->num;
+
+ /* Grab the next descriptor number they're advertising, and increment
+ * the index we've seen. */
+ if (get_user(head, &vq->avail->ring[last_avail_idx % vq->num])) {
+ vq_err(vq, "Failed to read head: idx %d address %p\n",
+ idx, &vq->avail->ring[last_avail_idx % vq->num]);
+ return vq->num;
+ }
+
+ /* If their number is silly, that's a fatal mistake. */
+ if (head >= vq->num) {
+ vq_err(vq, "Guest says index %u > %u is available",
+ head, vq->num);
+ return vq->num;
+ }
+
+ vq->last_avail_idx++;
+
+ /* When we start there are none of either input nor output. */
+ *out_num = *in_num = 0;
+
+ i = head;
+ do {
+ unsigned *num;
+ unsigned iov_count;
+ if (copy_from_user(&desc, vq->desc + i, sizeof desc)) {
+ vq_err(vq, "Failed to get descriptor: idx %d addr %p\n",
+ i, vq->desc + i);
+ return vq->num;
+ }
+ /* If this is an input descriptor, increment that count. */
+ if (desc.flags & VRING_DESC_F_WRITE) {
+ num = in_num;
+ iov_count = *out_num;
+ } else {
+ /* If it's an output descriptor, they're all supposed
+ * to come before any input descriptors. */
+ if (*in_num) {
+ vq_err(vq, "Descriptor has out after in: "
+ "idx %d\n", i);
+ return vq->num;
+ }
+ num = out_num;
+ iov_count = *in_num;
+ }
+ if (translate_desc(dev, desc.addr, desc.len, iov, iov_count,
+ VHOST_NET_MAX_SG, num)) {
+ vq_err(vq, "Failed to translate descriptor: idx %d\n",
+ i);
+ return vq->num;
+ }
+
+ /* If we've got too many, that implies a descriptor loop. */
+ if (*out_num + *in_num > vq->num) {
+ vq_err(vq, "Looped descriptor: idx %d\n", i);
+ return vq->num;
+ }
+ } while ((i = next_desc(vq, &desc)) != vq->num);
+
+ vq->inflight++;
+ return head;
+}
+
+/* Reverse the effect of vhost_get_vq_desc. Useful for error handling. */
+void vhost_discard_vq_desc(struct vhost_virtqueue *vq)
+{
+ vq->last_avail_idx--;
+ vq->inflight--;
+}
+
+/* After we've used one of their buffers, we tell them about it. We'll then
+ * want to send them an interrupt, using vq->call. */
+int vhost_add_used(struct vhost_virtqueue *vq,
+ unsigned int head, int len)
+{
+ struct vring_used_elem *used;
+
+ /* The virtqueue contains a ring of used buffers. Get a pointer to the
+ * next entry in that used ring. */
+ used = &vq->used->ring[vq->last_used_idx % vq->num];
+ if (put_user(head, &used->id)) {
+ vq_err(vq, "Failed to write used id");
+ return -EFAULT;
+ }
+ if (put_user(len, &used->len)) {
+ vq_err(vq, "Failed to write used len");
+ return -EFAULT;
+ }
+ /* Make sure buffer is written before we update index. */
+ wmb();
+ if (put_user(vq->last_used_idx + 1, &vq->used->idx)) {
+ vq_err(vq, "Failed to increment used idx");
+ return -EFAULT;
+ }
+ vq->last_used_idx++;
+ vq->inflight--;
+ return 0;
+}
+
+/* This actually sends the interrupt for this virtqueue */
+void vhost_trigger_irq(struct vhost_virtqueue *vq)
+{
+ __u16 flags = 0;
+ if (get_user(flags, &vq->avail->flags)) {
+ vq_err(vq, "Failed to get flags");
+ return;
+ }
+
+ /* If they don't want an interrupt, don't send one, unless empty. */
+ if ((flags & VRING_AVAIL_F_NO_INTERRUPT) && vq->inflight)
+ return;
+
+ /* Send the Guest an interrupt tell them we used something up. */
+ if (vq->call_ctx)
+ eventfd_signal(vq->call_ctx, 1);
+}
+
+/* And here's the combo meal deal. Supersize me! */
+void vhost_add_used_and_trigger(struct vhost_virtqueue *vq,
+ unsigned int head, int len)
+{
+ vhost_add_used(vq, head, len);
+ vhost_trigger_irq(vq);
+}
+
+int vhost_init(void)
+{
+ vhost_workqueue = create_workqueue("vhost");
+ if (!vhost_workqueue)
+ return -ENOMEM;
+ return 0;
+}
+
+void vhost_cleanup(void)
+{
+ destroy_workqueue(vhost_workqueue);
+}
diff --git a/drivers/vhost/vhost.h b/drivers/vhost/vhost.h
new file mode 100644
index 0000000..7f7ffcd
--- /dev/null
+++ b/drivers/vhost/vhost.h
@@ -0,0 +1,108 @@
+#ifndef _VHOST_H
+#define _VHOST_H
+
+#include <linux/eventfd.h>
+#include <linux/vhost.h>
+#include <linux/mm.h>
+#include <linux/mutex.h>
+#include <linux/workqueue.h>
+#include <linux/poll.h>
+#include <linux/file.h>
+#include <linux/skbuff.h>
+
+struct vhost_device;
+
+enum {
+ VHOST_NET_MAX_SG = MAX_SKB_FRAGS + 2,
+};
+
+/* Poll a file (eventfd or socket) */
+/* Note: there's nothing vhost specific about this structure. */
+struct vhost_poll {
+ poll_table table;
+ wait_queue_head_t *wqh;
+ wait_queue_t wait;
+ /* struct which will handle all actual work. */
+ struct work_struct work;
+ unsigned long mask;
+};
+
+void vhost_poll_init(struct vhost_poll *poll, work_func_t func,
+ unsigned long mask);
+void vhost_poll_start(struct vhost_poll *poll, struct file *file);
+void vhost_poll_stop(struct vhost_poll *poll);
+void vhost_poll_flush(struct vhost_poll *poll);
+
+/* The virtqueue structure describes a queue attached to a device. */
+struct vhost_virtqueue {
+ struct vhost_dev *dev;
+
+ /* The actual ring of buffers. */
+ struct mutex mutex;
+ unsigned int num;
+ struct vring_desc __user *desc;
+ struct vring_avail __user *avail;
+ struct vring_used __user *used;
+ struct file *kick;
+ struct file *call;
+ struct file *error;
+ struct eventfd_ctx *call_ctx;
+ struct eventfd_ctx *error_ctx;
+
+ struct vhost_poll poll;
+
+ /* The routine to call when the Guest pings us, or timeout. */
+ work_func_t handle_kick;
+
+ /* Last available index we saw. */
+ u16 last_avail_idx;
+
+ /* Last index we used. */
+ u16 last_used_idx;
+
+ /* Outstanding buffers */
+ unsigned int inflight;
+
+ /* Is this blocked? */
+ bool blocked;
+
+ struct iovec iov[VHOST_NET_MAX_SG];
+
+} ____cacheline_aligned;
+
+struct vhost_dev {
+ /* Readers use RCU to access memory table pointer.
+ * Writers use mutex below.*/
+ struct vhost_memory *memory;
+ struct mm_struct *mm;
+ struct vhost_virtqueue *vqs;
+ int nvqs;
+ struct mutex mutex;
+};
+
+long vhost_dev_init(struct vhost_dev *, struct vhost_virtqueue *vqs, int nvqs);
+long vhost_dev_check_owner(struct vhost_dev *);
+long vhost_dev_reset_owner(struct vhost_dev *);
+void vhost_dev_cleanup(struct vhost_dev *);
+long vhost_dev_ioctl(struct vhost_dev *, unsigned int ioctl, unsigned long arg);
+
+unsigned vhost_get_vq_desc(struct vhost_dev *, struct vhost_virtqueue *,
+ struct iovec iov[],
+ unsigned int *out_num, unsigned int *in_num);
+void vhost_discard_vq_desc(struct vhost_virtqueue *);
+
+int vhost_add_used(struct vhost_virtqueue *, unsigned int head, int len);
+void vhost_trigger_irq(struct vhost_virtqueue *);
+void vhost_add_used_and_trigger(struct vhost_virtqueue *,
+ unsigned int head, int len);
+
+int vhost_init(void);
+void vhost_cleanup(void);
+
+#define vq_err(vq, fmt, ...) do { \
+ printk(KERN_ERR pr_fmt(fmt), ##__VA_ARGS__); \
+ if ((vq)->error_ctx) \
+ eventfd_signal((vq)->error_ctx, 1);\
+ } while (0)
+
+#endif
diff --git a/include/linux/Kbuild b/include/linux/Kbuild
index dec2f18..975df9a 100644
--- a/include/linux/Kbuild
+++ b/include/linux/Kbuild
@@ -360,6 +360,7 @@ unifdef-y += uio.h
unifdef-y += unistd.h
unifdef-y += usbdevice_fs.h
unifdef-y += utsname.h
+unifdef-y += vhost.h
unifdef-y += videodev2.h
unifdef-y += videodev.h
unifdef-y += virtio_config.h
diff --git a/include/linux/miscdevice.h b/include/linux/miscdevice.h
index 0521177..781a8bb 100644
--- a/include/linux/miscdevice.h
+++ b/include/linux/miscdevice.h
@@ -30,6 +30,7 @@
#define HPET_MINOR 228
#define FUSE_MINOR 229
#define KVM_MINOR 232
+#define VHOST_NET_MINOR 233
#define MISC_DYNAMIC_MINOR 255
struct device;
diff --git a/include/linux/vhost.h b/include/linux/vhost.h
new file mode 100644
index 0000000..9ec6d5f
--- /dev/null
+++ b/include/linux/vhost.h
@@ -0,0 +1,100 @@
+#ifndef _LINUX_VHOST_H
+#define _LINUX_VHOST_H
+/* Userspace interface for in-kernel virtio accelerators. */
+
+/* vhost is used to reduce the number of system calls involved in virtio.
+ *
+ * Existing virtio net code is used in the guest without modification.
+ *
+ * This header includes interface used by userspace hypervisor for
+ * device configuration.
+ */
+
+#include <linux/types.h>
+#include <linux/compiler.h>
+#include <linux/ioctl.h>
+#include <linux/virtio_config.h>
+#include <linux/virtio_ring.h>
+
+struct vhost_vring_state {
+ unsigned int index;
+ unsigned int num;
+};
+
+struct vhost_vring_file {
+ unsigned int index;
+ int fd;
+};
+
+struct vhost_vring_addr {
+ unsigned int index;
+ unsigned int padding;
+ __u64 user_addr;
+};
+
+struct vhost_memory_region {
+ __u64 guest_phys_addr;
+ __u64 memory_size; /* bytes */
+ __u64 userspace_addr;
+ __u64 padding; /* read/write protection? */
+};
+
+struct vhost_memory {
+ __u32 nregions;
+ __u32 padding;
+ struct vhost_memory_region regions[0];
+};
+
+/* ioctls */
+
+#define VHOST_VIRTIO 0xAF
+
+/* Features bitmask for forward compatibility. Transport bits must be zero. */
+#define VHOST_GET_FEATURES _IOR(VHOST_VIRTIO, 0x00, __u32)
+#define VHOST_ACK_FEATURES _IOW(VHOST_VIRTIO, 0x00, __u32)
+
+/* Set current process as the (exclusive) owner of this file descriptor. This
+ * must be called before any other vhost command. Further calls to
+ * VHOST_OWNER_SET fail until VHOST_OWNER_RESET is called. */
+#define VHOST_SET_OWNER _IO(VHOST_VIRTIO, 0x01)
+/* Give up ownership, and reset the device to default values.
+ * Allows subsequent call to VHOST_OWNER_SET to succeed. */
+#define VHOST_RESET_OWNER _IO(VHOST_VIRTIO, 0x02)
+
+/* Set up/modify memory layout */
+#define VHOST_SET_MEM_TABLE _IOW(VHOST_VIRTIO, 0x03, struct vhost_memory)
+
+/* Ring setup. These parameters can not be modified while ring is running
+ * (bound to a device). */
+/* Set number of descriptors in ring */
+#define VHOST_SET_VRING_NUM _IOW(VHOST_VIRTIO, 0x10, struct vhost_vring_state)
+/* Start of array of descriptors (virtually contiguous) */
+#define VHOST_SET_VRING_DESC _IOW(VHOST_VIRTIO, 0x11, struct vhost_vring_addr)
+/* Used structure address */
+#define VHOST_SET_VRING_USED _IOW(VHOST_VIRTIO, 0x12, struct vhost_vring_addr)
+/* Available structure address */
+#define VHOST_SET_VRING_AVAIL _IOW(VHOST_VIRTIO, 0x13, struct vhost_vring_addr)
+/* Base value where queue looks for available descriptors */
+#define VHOST_SET_VRING_BASE _IOW(VHOST_VIRTIO, 0x14, struct vhost_vring_state)
+/* Get accessor: reads index, writes value in num */
+#define VHOST_GET_VRING_BASE _IOWR(VHOST_VIRTIO, 0x14, struct vhost_vring_state)
+
+/* The following ioctls use eventfd file descriptors to signal and poll
+ * for events. */
+
+/* Set eventfd to poll for added buffers */
+#define VHOST_SET_VRING_KICK _IOW(VHOST_VIRTIO, 0x20, struct vhost_vring_file)
+/* Set eventfd to signal when buffers have beed used */
+#define VHOST_SET_VRING_CALL _IOW(VHOST_VIRTIO, 0x21, struct vhost_vring_file)
+/* Set eventfd to signal an error */
+#define VHOST_SET_VRING_ERR _IOW(VHOST_VIRTIO, 0x22, struct vhost_vring_file)
+
+/* VHOST_NET specific defines */
+
+/* Attach virtio net device to a raw socket. The socket must be already
+ * bound to an ethernet device, this device will be used for transmit.
+ * Pass -1 to unbind from the socket and the transmit device.
+ * This can be used to stop the device (e.g. for migration). */
+#define VHOST_NET_SET_SOCKET _IOW(VHOST_VIRTIO, 0x30, int)
+
+#endif
--
1.6.2.5
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* [PATCHv4 1/2] mm: export use_mm/unuse_mm to modules
From: Michael S. Tsirkin @ 2009-08-19 15:02 UTC (permalink / raw)
To: netdev, virtualization, kvm, linux-kernel, mingo, linux-mm, hpa,
gr
In-Reply-To: <cover.1250693417.git.mst@redhat.com>
vhost net module wants to do copy to/from user from a kernel thread,
which needs use_mm (like what fs/aio has). Move that into mm/ and
export to modules.
Acked-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: Andrea Arcangeli <aarcange@redhat.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
fs/aio.c | 47 +----------------------------------
include/linux/mmu_context.h | 9 ++++++
mm/Makefile | 2 +-
mm/mmu_context.c | 58 +++++++++++++++++++++++++++++++++++++++++++
4 files changed, 69 insertions(+), 47 deletions(-)
create mode 100644 include/linux/mmu_context.h
create mode 100644 mm/mmu_context.c
diff --git a/fs/aio.c b/fs/aio.c
index d065b2c..fc21c23 100644
--- a/fs/aio.c
+++ b/fs/aio.c
@@ -24,6 +24,7 @@
#include <linux/file.h>
#include <linux/mm.h>
#include <linux/mman.h>
+#include <linux/mmu_context.h>
#include <linux/slab.h>
#include <linux/timer.h>
#include <linux/aio.h>
@@ -34,7 +35,6 @@
#include <asm/kmap_types.h>
#include <asm/uaccess.h>
-#include <asm/mmu_context.h>
#if DEBUG > 1
#define dprintk printk
@@ -595,51 +595,6 @@ static struct kioctx *lookup_ioctx(unsigned long ctx_id)
}
/*
- * use_mm
- * Makes the calling kernel thread take on the specified
- * mm context.
- * Called by the retry thread execute retries within the
- * iocb issuer's mm context, so that copy_from/to_user
- * operations work seamlessly for aio.
- * (Note: this routine is intended to be called only
- * from a kernel thread context)
- */
-static void use_mm(struct mm_struct *mm)
-{
- struct mm_struct *active_mm;
- struct task_struct *tsk = current;
-
- task_lock(tsk);
- active_mm = tsk->active_mm;
- atomic_inc(&mm->mm_count);
- tsk->mm = mm;
- tsk->active_mm = mm;
- switch_mm(active_mm, mm, tsk);
- task_unlock(tsk);
-
- mmdrop(active_mm);
-}
-
-/*
- * unuse_mm
- * Reverses the effect of use_mm, i.e. releases the
- * specified mm context which was earlier taken on
- * by the calling kernel thread
- * (Note: this routine is intended to be called only
- * from a kernel thread context)
- */
-static void unuse_mm(struct mm_struct *mm)
-{
- struct task_struct *tsk = current;
-
- task_lock(tsk);
- tsk->mm = NULL;
- /* active_mm is still 'mm' */
- enter_lazy_tlb(mm, tsk);
- task_unlock(tsk);
-}
-
-/*
* Queue up a kiocb to be retried. Assumes that the kiocb
* has already been marked as kicked, and places it on
* the retry run list for the corresponding ioctx, if it
diff --git a/include/linux/mmu_context.h b/include/linux/mmu_context.h
new file mode 100644
index 0000000..70fffeb
--- /dev/null
+++ b/include/linux/mmu_context.h
@@ -0,0 +1,9 @@
+#ifndef _LINUX_MMU_CONTEXT_H
+#define _LINUX_MMU_CONTEXT_H
+
+struct mm_struct;
+
+void use_mm(struct mm_struct *mm);
+void unuse_mm(struct mm_struct *mm);
+
+#endif
diff --git a/mm/Makefile b/mm/Makefile
index 5e0bd64..46c3892 100644
--- a/mm/Makefile
+++ b/mm/Makefile
@@ -11,7 +11,7 @@ obj-y := bootmem.o filemap.o mempool.o oom_kill.o fadvise.o \
maccess.o page_alloc.o page-writeback.o pdflush.o \
readahead.o swap.o truncate.o vmscan.o shmem.o \
prio_tree.o util.o mmzone.o vmstat.o backing-dev.o \
- page_isolation.o mm_init.o $(mmu-y)
+ page_isolation.o mm_init.o mmu_context.o $(mmu-y)
obj-y += init-mm.o
obj-$(CONFIG_PROC_PAGE_MONITOR) += pagewalk.o
diff --git a/mm/mmu_context.c b/mm/mmu_context.c
new file mode 100644
index 0000000..9989c2f
--- /dev/null
+++ b/mm/mmu_context.c
@@ -0,0 +1,58 @@
+/* Copyright (C) 2009 Red Hat, Inc.
+ *
+ * See ../COPYING for licensing terms.
+ */
+
+#include <linux/mm.h>
+#include <linux/mmu_context.h>
+#include <linux/module.h>
+#include <linux/sched.h>
+
+#include <asm/mmu_context.h>
+
+/*
+ * use_mm
+ * Makes the calling kernel thread take on the specified
+ * mm context.
+ * Called by the retry thread execute retries within the
+ * iocb issuer's mm context, so that copy_from/to_user
+ * operations work seamlessly for aio.
+ * (Note: this routine is intended to be called only
+ * from a kernel thread context)
+ */
+void use_mm(struct mm_struct *mm)
+{
+ struct mm_struct *active_mm;
+ struct task_struct *tsk = current;
+
+ task_lock(tsk);
+ active_mm = tsk->active_mm;
+ atomic_inc(&mm->mm_count);
+ tsk->mm = mm;
+ tsk->active_mm = mm;
+ switch_mm(active_mm, mm, tsk);
+ task_unlock(tsk);
+
+ mmdrop(active_mm);
+}
+EXPORT_SYMBOL_GPL(use_mm);
+
+/*
+ * unuse_mm
+ * Reverses the effect of use_mm, i.e. releases the
+ * specified mm context which was earlier taken on
+ * by the calling kernel thread
+ * (Note: this routine is intended to be called only
+ * from a kernel thread context)
+ */
+void unuse_mm(struct mm_struct *mm)
+{
+ struct task_struct *tsk = current;
+
+ task_lock(tsk);
+ tsk->mm = NULL;
+ /* active_mm is still 'mm' */
+ enter_lazy_tlb(mm, tsk);
+ task_unlock(tsk);
+}
+EXPORT_SYMBOL_GPL(unuse_mm);
--
1.6.2.5
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* Re: Kernel oops on setting sky2 interfaces down
From: Mike McCormack @ 2009-08-19 15:00 UTC (permalink / raw)
To: Rene Mayrhofer; +Cc: netdev, Richard Leitner, Stephen Hemminger
In-Reply-To: <200908190902.02854.rene.mayrhofer@gibraltar.at>
[-- Attachment #1: Type: text/plain, Size: 926 bytes --]
2009/8/19 Rene Mayrhofer <rene.mayrhofer@gibraltar.at>:
> Hi everybody,
>
> On Tuesday 11 August 2009 10:54:53 am Rene Mayrhofer wrote:
>> Thus, there really seems to be an uncaught case in sky2.c. When
>> sky2_phy_power_down is not called, chip should not go down, right? But
>> still sky2_poll seems to be called (maybe by an interrupt belonging to
>> another network interface but the same chip)?
>
> Is there anything else I could try? We still have this issue, making one range
> of hardware appliances unusable with 2.6 kernels...
Hi Rene,
There's a couple of things to try:
* try the latest sky2 code from net-next-2.6
* try adding an msleep(1) after sky2_rx_stop() in sky2_down()
* try adding a check for rx_ring and tx_ring being NULL in
sky2_status_intr(), and disable napi while freeing the buffers in
sky2_down()
I've got an untested, ad-hoc patch against net-next-2.6 for the last
two bits ...
thanks,
Mike
[-- Attachment #2: sky2-bandaids.diff --]
[-- Type: text/x-patch, Size: 1382 bytes --]
diff --git a/drivers/net/sky2.c b/drivers/net/sky2.c
index 8a763f8..a4c2778 100644
--- a/drivers/net/sky2.c
+++ b/drivers/net/sky2.c
@@ -1874,6 +1874,8 @@ static int sky2_down(struct net_device *dev)
sky2_rx_stop(sky2);
+ msleep(1);
+
/* Disable port IRQ */
imask = sky2_read32(hw, B0_IMSK);
imask &= ~portirq_msk[port];
@@ -1881,7 +1883,7 @@ static int sky2_down(struct net_device *dev)
sky2_read32(hw, B0_IMSK);
synchronize_irq(hw->pdev->irq);
- napi_synchronize(&hw->napi);
+ napi_disable(&hw->napi);
spin_lock_bh(&sky2->phy_lock);
sky2_phy_power_down(hw, port);
@@ -1912,6 +1914,8 @@ static int sky2_down(struct net_device *dev)
sky2->rx_ring = NULL;
sky2->tx_ring = NULL;
+ napi_enable(&hw->napi);
+
return 0;
}
@@ -2371,7 +2375,7 @@ static inline void sky2_tx_done(struct net_device *dev, u16 last)
{
struct sky2_port *sky2 = netdev_priv(dev);
- if (netif_running(dev))
+ if (likely(netif_running(dev) && sky2->tx_ring))
sky2_tx_complete(sky2, last);
}
@@ -2437,6 +2441,12 @@ static int sky2_status_intr(struct sky2_hw *hw, int to_do, u16 idx)
length = le16_to_cpu(le->length);
status = le32_to_cpu(le->status);
+ if (unlikely(!sky2->rx_ring)) {
+ printk(KERN_INFO PFX "%s: rx ring NULL %08x\n",
+ dev->name, opcode);
+ continue;
+ }
+
le->opcode = 0;
switch (opcode & ~HW_OWNER) {
case OP_RXSTAT:
^ permalink raw reply related
* [PATCHv4 0/2] vhost: a kernel-level virtio server
From: Michael S. Tsirkin @ 2009-08-19 15:00 UTC (permalink / raw)
To: netdev, virtualization, kvm, linux-kernel, mingo, linux-mm, akpm
Rusty, could you review and comment on the patches please? Since most
of the code deals with virtio from host side, I think it will make sense
to merge them through your tree. What do you think?
One comment on placement: I put files under a separate vhost directory
to avoid confusion with virtio-net which runs in guest. Does this sound
sane? If not let me know and I'll move them.
Thanks!
---
This implements vhost: a kernel-level backend for virtio,
The main motivation for this work is to reduce virtualization
overhead for virtio by removing system calls on data path,
without guest changes. For virtio-net, this removes up to
4 system calls per packet: vm exit for kick, reentry for kick,
iothread wakeup for packet, interrupt injection for packet.
This driver is as minimal as possible and does not implement any virtio
optional features, but it's fully functional (including migration
support), and already shows a latency improvement over userspace.
Some more detailed description attached to the patch itself.
The patches are against 2.6.31-rc4. I'd like them to go into linux-next
and down the road 2.6.32 if possible. Please comment.
Changelog from v3:
- checkpatch fixes
Changelog from v2:
- Comments on RCU usage
- Compat ioctl support
- Make variable static
- Copied more idiomatic english from Rusty
Changes from v1:
- Move use_mm/unuse_mm from fs/aio.c to mm instead of copying.
- Reorder code to avoid need for forward declarations
- Kill a couple of debugging printks
Michael S. Tsirkin (2):
mm: export use_mm/unuse_mm to modules
vhost_net: a kernel-level virtio server
MAINTAINERS | 10 +
arch/x86/kvm/Kconfig | 1 +
drivers/Makefile | 1 +
drivers/vhost/Kconfig | 11 +
drivers/vhost/Makefile | 2 +
drivers/vhost/net.c | 429 ++++++++++++++++++++++++++++
drivers/vhost/vhost.c | 664 +++++++++++++++++++++++++++++++++++++++++++
drivers/vhost/vhost.h | 108 +++++++
fs/aio.c | 47 +---
include/linux/Kbuild | 1 +
include/linux/miscdevice.h | 1 +
include/linux/mmu_context.h | 9 +
include/linux/vhost.h | 100 +++++++
mm/Makefile | 2 +-
mm/mmu_context.c | 58 ++++
15 files changed, 1397 insertions(+), 47 deletions(-)
create mode 100644 drivers/vhost/Kconfig
create mode 100644 drivers/vhost/Makefile
create mode 100644 drivers/vhost/net.c
create mode 100644 drivers/vhost/vhost.c
create mode 100644 drivers/vhost/vhost.h
create mode 100644 include/linux/mmu_context.h
create mode 100644 include/linux/vhost.h
create mode 100644 mm/mmu_context.c
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* (unknown),
From: Sumedha Gupta @ 2009-08-19 14:59 UTC (permalink / raw)
To: netdev
I have set up NIC bonding on a computer. NIC has four ports each with
a capacity of 1 Gbps. After configuring the switch, I am able to get
approximately 3.5 Gbps with round-robin, TLB and ALB and 3Gbps with
XOR. However when I remove bonding and give each port (eth0, eth1,
eth2, eth3) a different IP address and mac address and send data from
4 different client machines to different ports e.g first machine --->
eth0, second machine --->eth1, third machine ---->eth2 and fourth
machine---->eth3. With this configuration, how much bandwith should I
get? I think I should get 4 Gbps because I am using 4 different IP
address they should respond individually. However I am getting 1Gbps
total. Please correct if I am wrong and if I should get 4 Gbps then I
do I need to change something in configuration?
^ permalink raw reply
* 100Mbit ethernet performance on embedded devices
From: Johannes Stezenbach @ 2009-08-19 14:50 UTC (permalink / raw)
To: linux-embedded; +Cc: netdev
Hi,
a while ago I was working on a SoC with 200MHz ARM926EJ-S CPU
and integrated 100Mbit ethernet core, connected on internal
(fast) memory bus, with DMA. With iperf I measured:
TCP RX ~70Mbit/sec (iperf -s on SoC, iperf -c on destop PC)
TCP TX ~56Mbit/sec (iperf -s on destop PC, iperf -c o SoC)
The CPU load during the iperf test is around
1% user, 44% system, 4% irq, 48% softirq, with 7500 irqs/sec.
The kernel used in these measurements does not have iptables
support, I think packet filtering will slow it down noticably,
but I didn't actually try. The ethernet driver uses NAPI,
but it doesn't seem to be a win judging from the irq/sec number.
The kernel was an ancient 2.6.20.
I tried hard, but I couldn't find any performance figures for
comparison. (All performance figures I found refer to 1Gbit
or 10Gbit server type systems.)
What I'm interested in are some numbers for similar hardware,
to find out if my hardware and/or ethernet driver can be improved,
or if the CPU will always be the limiting factor.
I'd also be interested to know if hardware checksumming
support would improve throughput noticably in such a system,
or if it is only useful for 1Gbit and above.
Did anyone actually manage to get close to 100Mbit/sec
with similar CPU resources?
TIA,
Johannes
^ permalink raw reply
* Re: [PATCH 2/2] sctp: fix heartbeat count of path failure
From: Vlad Yasevich @ 2009-08-19 14:48 UTC (permalink / raw)
To: Chunbo Luo; +Cc: davem, netdev, linux-sctp
In-Reply-To: <1250665268-29770-2-git-send-email-chunbo.luo@windriver.com>
Chunbo Luo wrote:
> RFC4960 Section 8.2 defined that the transport should enter INACTIVE
> state only when the value in the error counter exceeds the protocol
> parameter 'Path.Max.Retrans' of that destination address. This means
> that the transport should enter INACTIVE state after pathmaxrxt+1
> heartbeats are not acknowledged.
>
>
> Signed-off-by: Chunbo Luo <chunbo.luo@windriver.com>
NAK. This patch seems to resurface periodically and I have to keep
explaining that it's wrong.
Every time we send a HB, we tick up the error count and clear it when
the HB-ACK is received. Each HB is separate and not a retransmission,
so we once we reach the pathmaxrxt, we've already sent max+1 HB, so we
have time out. Walk through the code with some values and you'll see
what I mean.
-vlad
> ---
> net/sctp/sm_sideeffect.c | 2 +-
> 1 files changed, 1 insertions(+), 1 deletions(-)
>
> diff --git a/net/sctp/sm_sideeffect.c b/net/sctp/sm_sideeffect.c
> index 86426aa..0e2e269 100644
> --- a/net/sctp/sm_sideeffect.c
> +++ b/net/sctp/sm_sideeffect.c
> @@ -447,7 +447,7 @@ static void sctp_do_8_2_transport_strike(struct sctp_association *asoc,
> asoc->overall_error_count++;
>
> if (transport->state != SCTP_INACTIVE &&
> - (transport->error_count++ >= transport->pathmaxrxt)) {
> + (transport->error_count++ > transport->pathmaxrxt)) {
> SCTP_DEBUG_PRINTK_IPADDR("transport_strike:association %p",
> " transport IP: port:%d failed.\n",
> asoc,
^ permalink raw reply
* Re: [PATCH v3 3/6] vbus: add a "vbus-proxy" bus model for vbus_driver objects
From: Avi Kivity @ 2009-08-19 14:35 UTC (permalink / raw)
To: Gregory Haskins
Cc: Ingo Molnar, Anthony Liguori, kvm, alacrityvm-devel, linux-kernel,
netdev, Michael S. Tsirkin, Ira W. Snyder
In-Reply-To: <4A8BFDD7.4040900@gmail.com>
On 08/19/2009 04:27 PM, Gregory Haskins wrote:
>> There's no inherent performance
>> problem in pci. The vbus approach has inherent problems (the biggest of
>> which is compatibility
>>
> Trying to be backwards compatible in all dimensions is not a design
> goal, as already stated.
>
It's important to me. If you ignore what's important to me don't expect
me to support your code.
>
> , the second managability).
>
>>
> Where are the management problems?
>
Requiring root, negotiation in the kernel (making it harder to set up a
compatible "migration pool", but wait, you don't care about migration
either.
> No, you have shown me that you disagree. I'm sorry, but do not assume
> they are the same.
[...]
> I'm sorry, but thats just plain false.
>
Don't you mean, "I disagree but that's completely different from you
being wrong".
>> Existing guests (Linux and
>> Windows) which support virtio will cease to work if the host moves to
>> vbus-virtio.
>>
> Sigh...please re-read "fact" section. And even if this work is accepted
> upstream as it is, how you configure the host and guest is just that: a
> configuration. If your guest and host both speak vbus, use it. If they
> don't, don't use it. Simple as that. Saying anything else is just more
> FUD, and I can say the same thing about a variety of other configuration
> options currently available.
>
The host, yes. The guest, no. I have RHEL 5.3 and Windows guests that
work with virtio now, and I'd like to keep it that way. Given that I
need to keep the current virtio-net/pci ABI, I have no motivation to add
other ABIs. Given that host userspace configuration works, I have no
motivation to move it into a kernel configfs/vbus based system. The
only thing that's hurting me is virtio-net's performance problems and
we're addressing it by moving the smallest possible component into the
kernel: vhost-net.
>> Existing hosts (running virtio-pci) won't be able to talk
>> to newer guests running virtio-vbus. The patch doesn't improve
>> performance without the entire vbus stack in the host kernel and a
>> vbus-virtio-net-host host kernel driver.
>>
> <rewind years=2>Existing hosts (running realtek emulation) won't be able
> to talk to newer guests running virtio-net. Virtio-net doesn't do
> anything to improve realtek emulation without the entire virtio stack in
> the host.</rewind>
>
> You gotta start somewhere. You're argument buys you nothing other than
> backwards compat, which I've already stated is not a specific goal here.
> I am not against "modprobe vbus-pcibridge", and I am sure there are
> users out that that do not object to this either.
>
Two years ago we had something that was set in stone and had a very
limited performance future. That's not the case now. If every two
years we start from scratch we'll be in a pretty pickle fairly soon.
virtio-net/pci is here to stay. I see no convincing reason to pour
efforts into a competitor and then have to support both.
>> Perhaps if you posted everything needed to make vbus-virtio work and
>> perform we could compare that to vhost-net and you'll see another reason
>> why vhost-net is the better approach.
>>
> Yet, you must recognize that an alternative outcome is that we can look
> at issues outside of virtio-net on KVM and perhaps you will see vbus is
> a better approach.
>
We won't know until that experiment takes place.
>>> You are also wrong to say that I didn't try to avoid creating a
>>> downstream effort first. I believe the public record of the mailing
>>> lists will back me up that I tried politely pushing this directly though
>>> kvm first. It was only after Avi recently informed me that they would
>>> be building their own version of an in-kernel backend in lieu of working
>>> with me to adapt vbus to their needs that I decided to put my own
>>> project together.
>>>
>>>
>> There's no way we can adapt vbus to our needs.
>>
> Really? Did you ever bother to ask how? I'm pretty sure you can. And
> if you couldn't, I would have considered changes to make it work.
>
Our needs are: compatibility, live migration, Windows, managebility
(nonroot, userspace control over configuration). Non-requirements but
highly desirable: minimal kernel impact.
>> Don't you think we'd preferred it rather than writing our own?
>>
> Honestly, I am not so sure based on your responses.
>
Does your experience indicate that I reject patches from others in
favour of writing my own?
Look for your own name in the kernel's git log.
> I've already listed numerous examples on why I advocate vbus over PCI,
> and have already stated I am not competing against virtio.
>
Well, your examples didn't convince me, and vbus's deficiencies
(compatibility, live migration, Windows, managebility, kernel impact)
aren't helping.
>> Showing some of those non-virt uses, for example.
>>
> Actually, Ira's chassis discussed earlier is a classic example. Vbus
> actually fits neatly into his model, I believe (and much better than the
> vhost proposals, IMO).
>
> Basically, IMO we want to invert Ira's bus (so that the PPC boards see
> host-based devices, instead of the other way around). You write a
> connector that transports the vbus verbs over the PCI link. You write a
> udev rule that responds to the PPC board "arrival" event to create a new
> vbus container, and assign the board to that context.
>
It's not inverted at all. vhost-net corresponds to the device side,
where a real NIC's DMA engine lives, while virtio-net is the guest side
which drives the device and talks only to its main memory (and device
registers). It may seem backwards but it's quite natural when you
consider DMA.
If you wish to push vbus for non-virt uses, I have nothing to say. If
you wish to push vbus for some other hypervisor (like AlacrityVM),
that's the other hypervisor's maintainer's turf. But vbus as I
understand it doesn't suit kvm's needs (compatibility, live migration,
Windows, managebility, kernel impact).
>> The fact that your only user duplicates existing functionality doesn't help.
>>
> Certainly at some level, that is true and is unfortunate, I agree. In
> retrospect, I wish I started with something non-overlapping with virtio
> as the demo, just to avoid this aspect of controversy.
>
> At another level, its the highest-performance 802.x interface for KVM at
> the moment, since we still have not seen benchmarks for vhost. Given
> that I have spent a lot of time lately optimizing KVM, I can tell you
> its not trivial to get it to work better than the userspace virtio.
> Michael is clearly a smart guy, so the odds are in his favor. But do
> not count your chickens before they hatch, because its not guaranteed
> success.
>
Well the latency numbers seem to match (after normalizing for host-host
baseline). Obviously throughput needs more work, but I have confidence
we'll see pretty good results.
> Long story short, my patches are not duplicative on all levels (i.e.
> performance). Its just another ethernet driver, of which there are
> probably hundreds of alternatives in the kernel already. You could also
> argue that we already have multiple models in qemu (realtek, e1000,
> virtio-net, etc) so this is not without precedent. So really all this
> "fragmentation" talk is FUD. Lets stay on-point, please.
>
It's not FUD and please talk technical, not throw words around. If
there are a limited number of kvm developers, then every new device
dilutes the effort. Further, e1000 and friends don't need drivers for a
bunch of OSs, v* do.
> Can we talk more about that at some point? I think you will see its not
> some "evil, heavy duty" infrastructure that some comments seem to be
> trying to paint it as. I think its similar in concept to what you need
> to do for a vhost like design, but (with all due respect to Michael) a
> little bit more thought into the necessary abstraction points to allow
> broader application.
>
vhost-net only pumps the rings. It leaves everything else for
userspace. vbus/venet leave almost nothing to userspace.
vbus redoes everything that the guest's native bus provides, virtio-pci
relies on pci. I haven't called it evil or heavy duty, just unnecessary.
(btw, your current alacrityvm patch is larger than kvm when it was first
merged into Linux)
>>>
>>>
>> Note whenever I mention migration, large guests, or Windows you say
>> these are not your design requirements.
>>
> Actually, I don't think I've ever said that, per se. I said that those
> things are not a priority for me, personally. I never made a design
> decision that I knew would preclude the support for such concepts. In
> fact, afaict, the design would support them just fine, given resources
> the develop them.
>
So given three choices:
1. merge vbus without those things that we need
2. merge vbus and start working on them
3. not merge vbus
As choice 1 gives me nothing and choice 2 takes away development effort,
choice 3 is the winner.
> For the record: I never once said "vbus is done". There is plenty of
> work left to do. This is natural (kvm I'm sure wasn't 100% when it went
> in either, nor is it today)
>
Which is why I want to concentrate effort in one direction, not wander
off in many.
--
error compiling committee.c: too many arguments to function
^ permalink raw reply
* Re: [PATCH v3 3/6] vbus: add a "vbus-proxy" bus model for vbus_driver objects
From: Michael S. Tsirkin @ 2009-08-19 14:33 UTC (permalink / raw)
To: Gregory Haskins
Cc: Avi Kivity, Anthony Liguori, Ingo Molnar, Gregory Haskins, kvm,
alacrityvm-devel, linux-kernel, netdev
In-Reply-To: <4A8B8F4E.80207@gmail.com>
On Wed, Aug 19, 2009 at 01:36:14AM -0400, Gregory Haskins wrote:
> Please post results when you have numbers, as I had to
> give up my 10GE rig in the lab.
> I suspect you will have performance
> issues until you at least address GSO, but you may already be there by now.
Yes, measuring streaming bandwidth probably does not make sense yet, as
I do not have GSO, and I do not have VM exit mitigation. But RSN.
Meanwhile udp_rr does not need any of these, so I checked that and numbers look
like what you'd expect. My systems seem slower than yours, but the
virtualization overhead is same: around 20us (sometimes it's a bit higher, up
to 25us).
host to host:
[root@virtlab18 netperf-2.4.5]# ~mst/netperf-2.4.5/bin/netperf -H 20.1.50.1 -t
+udp_rr
UDP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 20.1.50.1
+(20.1.50.1) port 0 AF_INET
Local /Remote
Socket Size Request Resp. Elapsed Trans.
Send Recv Size Size Time Rate
bytes Bytes bytes bytes secs. per sec
262144 262144 1 1 10.00 13890.41
124928 124928
host to guest:
[root@virtlab18 linux-2.6]# ~mst/netperf-2.4.5/bin/netperf -H 20.1.50.3 -t udp_rr
UDP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 20.1.50.3
+(20.1.50.3) port 0 AF_INET
Local /Remote
Socket Size Request Resp. Elapsed Trans.
Send Recv Size Size Time Rate
bytes Bytes bytes bytes secs. per sec
262144 262144 1 1 10.00 10884.78
124928 124928
--
MST
^ permalink raw reply
* Re: [RFC PATCH] lib/vsprintf.c: Add struct sockaddr * "%pN<foo>" output
From: Chuck Lever @ 2009-08-19 14:26 UTC (permalink / raw)
To: Joe Perches; +Cc: jens, brian.haley, David Miller, netdev
In-Reply-To: <1250395818.4625.15.camel@Joe-Laptop.home>
On Aug 16, 2009, at 12:10 AM, Joe Perches wrote:
> Hi Chuck.
>
> Here's a tentative patch that adds that facility you wanted in
> this thread.
> http://kerneltrap.org/mailarchive/linux-netdev/2008/11/25/4231684
>
> This patch is on top of this patch:
> http://marc.info/?l=linux-netdev&m=125034992003220&w=2
>
> I'm not sure it's great or even useful, but just for discussion.
I think having at least a generic socket address formatter would be
very helpful. The kernel RPC code can use that immediately for
generating debugging messages, generating "universal
addresses" (patches already accepted for 2.6.32) and for generating
presentation addresses to pass to user space (lockd does this now).
> Use style:
> * - 'N' For network socket (sockaddr) pointers
> * if sa_family is IPv4, output is %pI4; if IPv6, output is %pI6c
> * May be used with any combination of additional specifiers
> below
> * 'p' decimal socket port number for IPv[46]: ":12345"
> * 'f' decimal flowinfo for IPv6: "/123456789"
> * 's' decimal scope_id number for IPv6: "%1234567890"
> * so %pNp will print if IPv4 "1.2.3.4:1234", if IPv6:
> "1::c0a8:a:1234"
>
> I think using ":" as the separator for the port number, while common,
> and already frequently used in kernel source (see bullet 2 in):
> http://www.ietf.org/id/draft-kawamura-ipv6-text-representation-03.txt
> "Section 6: Notes on Combining IPv6 Addresses with Port Numbers".
> is not good for readability.
>
> Perhaps this style should be changed to the "[ipv6]:port" described
> in the draft above.
I agree that the port number convention is tricky, and some kernel
areas handle it differently than others. Perhaps having a separate
family-agnostic formatter for printing the port number (or scope ID,
etc.) might allow greatest flexibility.
The RPC code, for example, displays port numbers in decimal and in
"hi.lo" format (the high byte is printed in decimal, then the low byte
is printed in decimal. The two values are separated by a dot. For
example, port 2049 is displayed as "8.1").
> cheers, Joe
>
> diff --git a/lib/vsprintf.c b/lib/vsprintf.c
> index 9b79536..b3cbc38 100644
> --- a/lib/vsprintf.c
> +++ b/lib/vsprintf.c
> @@ -791,6 +791,90 @@ static char *ip4_addr_string(char *buf, char
> *end, const u8 *addr,
> return string(buf, end, ip4_addr, spec);
> }
>
> +static char *u32_dec_val(char *p, u32 val)
> +{
> + char temp[9];
> + int digits;
> + u32 hi_val = val / 100000;
> + char *pos;
> + pos = put_dec_trunc(temp, val%100000);
> + if (hi_val)
> + pos = put_dec_trunc(pos, hi_val);
> + digits = pos - temp;
> + /* reverse the digits in temp */
> + while (digits--)
> + *p++ = temp[digits];
> + return p;
> +}
> +
> +static char *socket_addr_string(char *buf, char *end,
> + const struct sockaddr *sa,
> + struct printf_spec spec, const char *fmt)
> +{
> + char addr[sizeof("xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:255.255.255.255") +
> + sizeof(":12345") +
> + sizeof("%1234567890") +
> + sizeof("/123456789")];
> + char *p;
> + struct sockaddr_in *sa4 = (struct sockaddr_in *)sa;
> + struct sockaddr_in6 *sa6 = (struct sockaddr_in6 *)sa;
> +
> + switch (sa->sa_family) {
> + case AF_INET:
> + p = ip4_string(addr, (const u8 *)&sa4->sin_addr.s_addr, false);
> + break;
> + case AF_INET6:
> + p = ip6_compressed_string(addr, &sa6->sin6_addr);
> + break;
> + default: {
> + struct printf_spec num_spec = {
> + .base = 16,
> + .precision = -1,
> + .field_width = 2 * sizeof(void *),
> + .flags = SPECIAL | SMALL | ZEROPAD,
> + };
> +
> + p = strcpy(addr, "Bad socket address: ")
> + + sizeof("Bad socket address: ");
> + p = number(p, addr + sizeof(addr), (unsigned long)sa, num_spec);
> + break;
> + }
> + }
> +
> + while (isalpha(*++fmt)) {
> + switch (*fmt) {
> + case 'p':
> + *p++ = ':';
> + switch (sa->sa_family) {
> + case AF_INET:
> + p = u32_dec_val(p,ntohs(sa4->sin_port));
> + break;
> + case AF_INET6:
> + p = u32_dec_val(p,ntohs(sa6->sin6_port));
> + break;
> + }
> + break;
> + case 's':
> + *p++ = '%';
> + switch (sa->sa_family) {
> + case AF_INET6:
> + p = u32_dec_val(p, sa6->sin6_scope_id);
> + }
> + break;
> + case 'f':
> + *p++ = '/';
> + switch (sa->sa_family) {
> + case AF_INET6:
> + p = u32_dec_val(p, ntohl(sa6->sin6_flowinfo &
> + IPV6_FLOWINFO_MASK));
> + }
> + break;
> + }
> + }
> + *p = '\0';
> + return string(buf, end, addr, spec);
> +}
> +
> /*
> * Show a '%p' thing. A kernel extension is that the '%p' is followed
> * by an extra set of alphanumeric characters that are extended format
> @@ -814,6 +898,13 @@ static char *ip4_addr_string(char *buf, char
> *end, const u8 *addr,
> * IPv4 uses dot-separated decimal with leading 0's
> (010.123.045.006)
> * - 'I6c' for IPv6 addresses printed as specified by
> * http://www.ietf.org/id/draft-kawamura-ipv6-text-representation-03.txt
> + * - 'N' For network socket (sockaddr) pointers
> + * if sa_family is IPv4, output is %pI4; if IPv6, output is
> %pI6c
> + * May be used with any combination of additional specifiers
> below
> + * 'p' decimal socket port number for IPv[46]: ":12345"
> + * 'f' decimal flowinfo for IPv6: "/123456789"
> + * 's' decimal scope_id number for IPv6: "%1234567890"
> + * so %pNp will print if IPv4 "1.2.3.4:1234", if IPv6:
> "1::c0a8:a:1234"
> * Note: The difference between 'S' and 'F' is that on ia64 and ppc64
> * function pointers are really function descriptors, which contain a
> * pointer to the real address.
> @@ -852,7 +943,10 @@ static char *pointer(const char *fmt, char
> *buf, char *end, void *ptr,
> return ip4_addr_string(buf, end, ptr, spec, fmt);
> }
> break;
> + case 'N':
> + return socket_addr_string(buf, end, ptr, spec, fmt);
> }
> +
> spec.flags |= SMALL;
> if (spec.field_width == -1) {
> spec.field_width = 2*sizeof(void *);
>
>
--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
^ permalink raw reply
* Re: [PATCHv3 2/2] vhost_net: a kernel-level virtio server
From: Michael S. Tsirkin @ 2009-08-19 14:20 UTC (permalink / raw)
To: Arnd Bergmann
Cc: virtualization, netdev, kvm, linux-kernel, mingo, linux-mm, akpm,
hpa, gregory.haskins, Or Gerlitz
In-Reply-To: <200908191546.44193.arnd@arndb.de>
On Wed, Aug 19, 2009 at 03:46:44PM +0200, Arnd Bergmann wrote:
> On Wednesday 19 August 2009, Michael S. Tsirkin wrote:
> > > Maybe we could instead extend the 'splice' system call to work on a
> > > vhost_net file descriptor. If we do that, we can put the access back
> > > into a user thread (or two) that stays in splice indefinetely
> >
> > An issue with exposing internal threading model to userspace
> > in this way is that we lose control of e.g. CPU locality -
> > and it is very hard for userspace to get it right.
>
> Good point, I hadn't thought about that in this context.
>
> For macvtap, my idea was to open the same tap char device multiple
> times and use each fd exclusively on one *guest* CPU. I'm not sure
> if virtio-net can already handle SMP guests efficiently. We might
> actually need to extend it to have more pairs of virtqueues, one
> for each guest CPU, which can then be bound to a host queue (or queue
> pair) in the physical nic.
>
> Leaving that aside for now, you could replace VHOST_NET_SET_SOCKET,
> VHOST_SET_OWNER, VHOST_RESET_OWNER
SET/RESET OWNER is still needed: otherwise if you share a descriptor
with another process, it can corrupt your memory.
> and your kernel thread with a new
> VHOST_NET_SPLICE blocking ioctl that does all the transfers in the
> context of the calling thread.
For one, you'd want a thread per virtqueue. Second, an incoming traffic
might arrive on another CPU, we want to keep it local. I guess you
would also want ioctls to wake up the threads spuriously ...
> This would improve the driver on various fronts:
>
> - no need for playing tricks with use_mm/unuse_mm
> - possibly fewer global TLB flushes from switch_mm, which
> may improve performance.
Why would there be less flushes?
> - ability to pass down error codes from socket or guest to
> user space by returning from ioctl
virtio can not pass error codes. translation errors are
simple enough to just have a counter.
> - based on that, the ability to use any kind of file
> descriptor that can do writev/readv or sendmsg/recvmsg
> without the nastiness you mentioned.
Yes, it's an interesting approach. As I said, need to tread very
carefully though, I don't think all issues are figured out. For example:
what happens if we pass our own fd here? Will refcount on file ever get
to 0 on exit? There may be others ...
> The disadvantage of course is that you need to add a user
> thread for each guest device to make up for the workqueue
> that you save.
More importantly, you lose control of CPU locality. Simply put, a
natural threading model in virtualization is one thread per guest vcpu.
Asking applications to add multiple helper threads just so they can
block forever is wrong, IMO, as userspace has no idea which CPU
they should be on, what priority to use, etc.
> > > to
> > > avoid some of the implications of kernel threads like the missing
> > > ability to handle transfer errors in user space.
> >
> > Are you talking about TCP here?
> > Transfer errors are typically asynchronous - possibly eventfd
> > as I expose for vhost net is sufficient there.
>
> I mean errors in general if we allow random file descriptors to be used.
> E.g. tun_chr_aio_read could return EBADFD, EINVAL, EFAULT, ERESTARTSYS,
> EIO, EAGAIN and possibly others. We can handle some in kernel, others
> should never happen with vhost_net, but if something unexpected happens
> it would be nice to just bail out to user space.
And note that there might be more than one error. I guess, that's
another problem with trying to layer on top of vfs.
> > > > I wonder - can we expose the underlying socket used by tap, or will that
> > > > create complex lifetime issues?
> > >
> > > I think this could get more messy in the long run than calling vfs_readv
> > > on a random fd. It would mean deep internal knowledge of the tap driver
> > > in vhost_net, which I really would prefer to avoid.
> >
> > No, what I had in mind is adding a GET_SOCKET ioctl to tap.
> > vhost would then just use the socket.
>
> Right, that would work with tun/tap at least. It sounds a bit fishy
> but I can't see a reason why it would be hard to do.
> I'd have to think about how to get it working with macvtap, or if
> there is much value left in macvtap after that anyway.
>
> > > So how about making the qemu command line interface an extension to
> > > what Or Gerlitz has done for the raw packet sockets?
> >
> > Not sure I see the connection, but I have not thought about qemu
> > side of things too much yet - trying to get kernel bits in place
> > first so that there's a stable ABI to work with.
>
> Ok, fair enough. The kernel bits are obviously more time critical
> right now, since they should get into 2.6.32.
>
> Arnd <><
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCHv3 2/2] vhost_net: a kernel-level virtio server
From: Arnd Bergmann @ 2009-08-19 13:46 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: virtualization, netdev, kvm, linux-kernel, mingo, linux-mm, akpm,
hpa, gregory.haskins, Or Gerlitz
In-Reply-To: <20090819130417.GB3080@redhat.com>
On Wednesday 19 August 2009, Michael S. Tsirkin wrote:
> > Maybe we could instead extend the 'splice' system call to work on a
> > vhost_net file descriptor. If we do that, we can put the access back
> > into a user thread (or two) that stays in splice indefinetely
>
> An issue with exposing internal threading model to userspace
> in this way is that we lose control of e.g. CPU locality -
> and it is very hard for userspace to get it right.
Good point, I hadn't thought about that in this context.
For macvtap, my idea was to open the same tap char device multiple
times and use each fd exclusively on one *guest* CPU. I'm not sure
if virtio-net can already handle SMP guests efficiently. We might
actually need to extend it to have more pairs of virtqueues, one
for each guest CPU, which can then be bound to a host queue (or queue
pair) in the physical nic.
Leaving that aside for now, you could replace VHOST_NET_SET_SOCKET,
VHOST_SET_OWNER, VHOST_RESET_OWNER and your kernel thread with a new
VHOST_NET_SPLICE blocking ioctl that does all the transfers in the
context of the calling thread.
This would improve the driver on various fronts:
- no need for playing tricks with use_mm/unuse_mm
- possibly fewer global TLB flushes from switch_mm, which
may improve performance.
- ability to pass down error codes from socket or guest to
user space by returning from ioctl
- based on that, the ability to use any kind of file
descriptor that can do writev/readv or sendmsg/recvmsg
without the nastiness you mentioned.
The disadvantage of course is that you need to add a user
thread for each guest device to make up for the workqueue
that you save.
> > to
> > avoid some of the implications of kernel threads like the missing
> > ability to handle transfer errors in user space.
>
> Are you talking about TCP here?
> Transfer errors are typically asynchronous - possibly eventfd
> as I expose for vhost net is sufficient there.
I mean errors in general if we allow random file descriptors to be used.
E.g. tun_chr_aio_read could return EBADFD, EINVAL, EFAULT, ERESTARTSYS,
EIO, EAGAIN and possibly others. We can handle some in kernel, others
should never happen with vhost_net, but if something unexpected happens
it would be nice to just bail out to user space.
> > > I wonder - can we expose the underlying socket used by tap, or will that
> > > create complex lifetime issues?
> >
> > I think this could get more messy in the long run than calling vfs_readv
> > on a random fd. It would mean deep internal knowledge of the tap driver
> > in vhost_net, which I really would prefer to avoid.
>
> No, what I had in mind is adding a GET_SOCKET ioctl to tap.
> vhost would then just use the socket.
Right, that would work with tun/tap at least. It sounds a bit fishy
but I can't see a reason why it would be hard to do.
I'd have to think about how to get it working with macvtap, or if
there is much value left in macvtap after that anyway.
> > So how about making the qemu command line interface an extension to
> > what Or Gerlitz has done for the raw packet sockets?
>
> Not sure I see the connection, but I have not thought about qemu
> side of things too much yet - trying to get kernel bits in place
> first so that there's a stable ABI to work with.
Ok, fair enough. The kernel bits are obviously more time critical
right now, since they should get into 2.6.32.
Arnd <><
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox