* Re: [PATCH v3 3/6] vbus: add a "vbus-proxy" bus model for
From: Gregory Haskins @ 2009-08-18 12:24 UTC (permalink / raw)
To: avi
Cc: mingo, gregory.haskins, alacrityvm-devel, mst, kvm, linux-kernel,
netdev
In-Reply-To: <4A8A65540200005A000528EA@sinclair.provo.novell.com>
(Again on the top post)
No, Avi, nothing has changed to my knowledge. I just saw that you and Michael were heading down the same path, so I thought I might interject that we've already covered that ground.
As of right now, I am of the opinion that its not worth any change in the short term, and may be worth IOoHC in the long term (primarily so that mmios get a boost)
-greg.
-----Original Message-----
From: Avi Kivity <avi@redhat.com>
To: Gregory Haskins <GHaskins@novell.com>
Cc: <mingo@elte.hu>
Cc: <gregory.haskins@gmail.com>
Cc: <alacrityvm-devel@lists.sourceforge.net>
Cc: <mst@redhat.com>
Cc: <kvm@vger.kernel.org>
Cc: <linux-kernel@vger.kernel.org>
Cc: <netdev@vger.kernel.org>
Sent: 8/18/2009 6:19:53 AM
Subject: Re: [PATCH v3 3/6] vbus: add a "vbus-proxy" bus model for vbus_drive
On 08/18/2009 03:11 PM, Gregory Haskins wrote:
> Sorry for the toppost. Still not at the office.
>
> I just wanted to add that we've already been through this disussion once. (Search "haskins hypercall lkml" on google and I'm sure you are bound to see hits.
>
>
Your numbers showed a 350ns difference on fairly old (by now) hardware.
I doubt the difference will exceed 200ns now.
> The fact is: the original vbus was designed with hypercalls, and it drew much of these same critisims. In the end, hypercalls are only marginally faster than PIO (iirc, 450ns faster, and shrinking), so we decided that it was not worth further discussion at the time.
>
Has anything changed?
> A better solution is probably PIOoHC, so that you retain the best properties of both. The only problem with the entire PIOx approach is that its x86 specific, but that is an entirely different thread.
>
pio is nicely abstracted by PCI. virtio-pci will use pio on x86 and
mmio on non-x86.
--
error compiling committee.c: too many arguments to function
^ permalink raw reply
* Re: [PATCH v3 3/6] vbus: add a "vbus-proxy" bus model for vbus_drive
From: Avi Kivity @ 2009-08-18 12:19 UTC (permalink / raw)
To: Gregory Haskins
Cc: mst, mingo, gregory.haskins, alacrityvm-devel, kvm, linux-kernel,
netdev
In-Reply-To: <4A8A62370200005A000528DF@sinclair.provo.novell.com>
On 08/18/2009 03:11 PM, Gregory Haskins wrote:
> Sorry for the toppost. Still not at the office.
>
> I just wanted to add that we've already been through this disussion once. (Search "haskins hypercall lkml" on google and I'm sure you are bound to see hits.
>
>
Your numbers showed a 350ns difference on fairly old (by now) hardware.
I doubt the difference will exceed 200ns now.
> The fact is: the original vbus was designed with hypercalls, and it drew much of these same critisims. In the end, hypercalls are only marginally faster than PIO (iirc, 450ns faster, and shrinking), so we decided that it was not worth further discussion at the time.
>
Has anything changed?
> A better solution is probably PIOoHC, so that you retain the best properties of both. The only problem with the entire PIOx approach is that its x86 specific, but that is an entirely different thread.
>
pio is nicely abstracted by PCI. virtio-pci will use pio on x86 and
mmio on non-x86.
--
error compiling committee.c: too many arguments to function
^ permalink raw reply
* Re: [PATCH v3 3/6] vbus: add a "vbus-proxy" bus model for vbus_drive
From: Gregory Haskins @ 2009-08-18 12:11 UTC (permalink / raw)
To: avi, mst
Cc: mingo, gregory.haskins, alacrityvm-devel, kvm, linux-kernel,
netdev
In-Reply-To: <4A8A62370200005A000528DF@sinclair.provo.novell.com>
Sorry for the toppost. Still not at the office.
I just wanted to add that we've already been through this disussion once. (Search "haskins hypercall lkml" on google and I'm sure you are bound to see hits.
The fact is: the original vbus was designed with hypercalls, and it drew much of these same critisims. In the end, hypercalls are only marginally faster than PIO (iirc, 450ns faster, and shrinking), so we decided that it was not worth further discussion at the time.
A better solution is probably PIOoHC, so that you retain the best properties of both. The only problem with the entire PIOx approach is that its x86 specific, but that is an entirely different thread.
Hth
-greg
-----Original Message-----
From: Avi Kivity <avi@redhat.com>
Cc: Gregory Haskins <GHaskins@novell.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Gregory Haskins <gregory.haskins@gmail.com>
Cc: <alacrityvm-devel@lists.sourceforge.net>
To: Michael S. Tsirkin <mst@redhat.com>
Cc: <kvm@vger.kernel.org>
Cc: <linux-kernel@vger.kernel.org>
Cc: <netdev@vger.kernel.org>
Sent: 8/18/2009 5:54:46 AM
Subject: Re: [PATCH v3 3/6] vbus: add a "vbus-proxy" bus model for vbus_driver objects
On 08/18/2009 02:49 PM, Michael S. Tsirkin wrote:
>
>> The host kernel sees a hypercall vmexit. How does it know if it's a
>> nested-guest-to-guest hypercall or a nested-guest-to-host hypercall?
>> The two are equally valid at the same time.
>>
> Here is how this can work - it is similar to MSI if you like:
> - by default, the device uses pio kicks
> - nested guest driver can enable hypercall capability in the device,
> probably with pci config cycle
> - guest userspace (hypervisor running in guest) will see this request
> and perform pci config cycle on the "real" device, telling it to which
> nested guest this device is assigned
>
So far so good.
> - host userspace (hypervisor running in host) will see this.
> it now knows both which guest the hypercalls will be for,
> and that the device in question is an emulated one,
> and can set up kvm appropriately
>
No it doesn't. The fact that one device uses hypercalls doesn't mean
all hypercalls are for that device. Hypercalls are a shared resource,
and there's no way to tell for a given hypercall what device it is
associated with (if any).
>> The host knows whether the guest or nested guest are running. If the
>> guest is running, it's a guest-to-host hypercall. If the nested guest
>> is running, it's a nested-guest-to-guest hypercall. We don't have
>> nested-guest-to-host hypercalls (and couldn't unless we get agreement on
>> a protocol from all hypervisor vendors).
>>
> Not necessarily. What I am saying is we could make this protocol part of
> guest paravirt driver. the guest that loads the driver and enables the
> capability, has to agree to the protocol. If it doesn't want to, it does
> not have to use that driver.
>
It would only work for kvm-on-kvm.
--
error compiling committee.c: too many arguments to function
^ permalink raw reply
* Re: Receive side performance issue with multi-10-GigE and NUMA
From: Andrew Gallatin @ 2009-08-18 11:54 UTC (permalink / raw)
To: Bill Fink
Cc: Jesse Barnes, Brandeburg, Jesse, Neil Horman, Brice Goglin,
Linux Network Developers, Yinghai Lu
In-Reply-To: <20090818030718.aef199f4.billfink@mindspring.com>
Bill Fink wrote:
> < Latency: 0, Cache Line Size: 64 bytes
<...>
>> Latency: 0, Cache Line Size: 256 bytes
A cache line size of 256 clearly seems wrong for a Xeon. I assume all
devices on the SuperMicro show the same value?
> Interestingly, the "WC Enabled" is only indicated on the first two
The WC is probably a red herring.
What does ethtool -S show for the DMA write bandwidth of the
NICs on the SuperMicro?
These values are obtained serially, as the driver resets
the NIC (reset happens at load time, and ifconfig up),
so they could easily sum to more than the memory bandwidth
of the system. But it would be good to check for any anomalies.
I can send you a pointer to a tool we use internally, which loads
some custom firmware on the NIC, and can exercise the DMA engines
on all the NICs in parallel. This would give an idea of the
aggregate DMA bandwidth available on the system. Let me know
if you're interested.
Drew
^ permalink raw reply
* Re: [PATCH v3 3/6] vbus: add a "vbus-proxy" bus model for vbus_driver objects
From: Avi Kivity @ 2009-08-18 11:54 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Gregory Haskins, Ingo Molnar, Gregory Haskins, kvm,
alacrityvm-devel, linux-kernel, netdev
In-Reply-To: <20090818114914.GA17721@redhat.com>
On 08/18/2009 02:49 PM, Michael S. Tsirkin wrote:
>
>> The host kernel sees a hypercall vmexit. How does it know if it's a
>> nested-guest-to-guest hypercall or a nested-guest-to-host hypercall?
>> The two are equally valid at the same time.
>>
> Here is how this can work - it is similar to MSI if you like:
> - by default, the device uses pio kicks
> - nested guest driver can enable hypercall capability in the device,
> probably with pci config cycle
> - guest userspace (hypervisor running in guest) will see this request
> and perform pci config cycle on the "real" device, telling it to which
> nested guest this device is assigned
>
So far so good.
> - host userspace (hypervisor running in host) will see this.
> it now knows both which guest the hypercalls will be for,
> and that the device in question is an emulated one,
> and can set up kvm appropriately
>
No it doesn't. The fact that one device uses hypercalls doesn't mean
all hypercalls are for that device. Hypercalls are a shared resource,
and there's no way to tell for a given hypercall what device it is
associated with (if any).
>> The host knows whether the guest or nested guest are running. If the
>> guest is running, it's a guest-to-host hypercall. If the nested guest
>> is running, it's a nested-guest-to-guest hypercall. We don't have
>> nested-guest-to-host hypercalls (and couldn't unless we get agreement on
>> a protocol from all hypervisor vendors).
>>
> Not necessarily. What I am saying is we could make this protocol part of
> guest paravirt driver. the guest that loads the driver and enables the
> capability, has to agree to the protocol. If it doesn't want to, it does
> not have to use that driver.
>
It would only work for kvm-on-kvm.
--
error compiling committee.c: too many arguments to function
^ permalink raw reply
* Re: [PATCH v3 3/6] vbus: add a "vbus-proxy" bus model for vbus_driver objects
From: Michael S. Tsirkin @ 2009-08-18 11:49 UTC (permalink / raw)
To: Avi Kivity
Cc: Gregory Haskins, Ingo Molnar, Gregory Haskins, kvm,
alacrityvm-devel, linux-kernel, netdev
In-Reply-To: <4A8A8D6D.9040909@redhat.com>
On Tue, Aug 18, 2009 at 02:15:57PM +0300, Avi Kivity wrote:
> On 08/18/2009 02:07 PM, Michael S. Tsirkin wrote:
>> On Tue, Aug 18, 2009 at 01:45:05PM +0300, Avi Kivity wrote:
>>
>>> On 08/18/2009 01:28 PM, Michael S. Tsirkin wrote:
>>>
>>>>
>>>>> Suppose a nested guest has two devices. One a virtual device backed by
>>>>> its host (our guest), and one a virtual device backed by us (the real
>>>>> host), and assigned by the guest to the nested guest. If both devices
>>>>> use hypercalls, there is no way to distinguish between them.
>>>>>
>>>>>
>>>> Not sure I understand. What I had in mind is that devices would have to
>>>> either use different hypercalls and map hypercall to address during
>>>> setup, or pass address with each hypercall. We get the hypercall,
>>>> translate the address as if it was pio access, and know the destination?
>>>>
>>>>
>>> There are no different hypercalls. There's just one hypercall
>>> instruction, and there's no standard on how it's used. If a nested call
>>> issues a hypercall instruction, you have no idea if it's calling a
>>> Hyper-V hypercall or a vbus/virtio kick.
>>>
>> userspace will know which it is, because hypercall capability
>> in the device has been activated, and can tell kernel, using
>> something similar to iosignalfd. No?
>>
>
> The host kernel sees a hypercall vmexit. How does it know if it's a
> nested-guest-to-guest hypercall or a nested-guest-to-host hypercall?
> The two are equally valid at the same time.
Here is how this can work - it is similar to MSI if you like:
- by default, the device uses pio kicks
- nested guest driver can enable hypercall capability in the device,
probably with pci config cycle
- guest userspace (hypervisor running in guest) will see this request
and perform pci config cycle on the "real" device, telling it to which
nested guest this device is assigned
- host userspace (hypervisor running in host) will see this.
it now knows both which guest the hypercalls will be for,
and that the device in question is an emulated one,
and can set up kvm appropriately
>>> You could have a protocol where you register the hypercall instruction's
>>> address with its recipient, but it quickly becomes a tangled mess.
>>>
>> I really thought we could pass the io address in register as an input
>> parameter. Is there a way to do this in a secure manner?
>>
>> Hmm. Doesn't kvm use hypercalls now? How does this work with nesting?
>> For example, in this code in arch/x86/kvm/x86.c:
>>
>> switch (nr) {
>> case KVM_HC_VAPIC_POLL_IRQ:
>> ret = 0;
>> break;
>> case KVM_HC_MMU_OP:
>> r = kvm_pv_mmu_op(vcpu, a0, hc_gpa(vcpu, a1, a2),&ret);
>> break;
>> default:
>> ret = -KVM_ENOSYS;
>> break;
>> }
>>
>> how do we know that it's the guest and not the nested guest performing
>> the hypercall?
>>
>
> The host knows whether the guest or nested guest are running. If the
> guest is running, it's a guest-to-host hypercall. If the nested guest
> is running, it's a nested-guest-to-guest hypercall. We don't have
> nested-guest-to-host hypercalls (and couldn't unless we get agreement on
> a protocol from all hypervisor vendors).
Not necessarily. What I am saying is we could make this protocol part of
guest paravirt driver. the guest that loads the driver and enables the
capability, has to agree to the protocol. If it doesn't want to, it does
not have to use that driver.
> --
> error compiling committee.c: too many arguments to function
^ permalink raw reply
* [net-next-2.6 PATCH] net/ethtool: Add support for the ethtool feature to flash firmware image from a specified file.
From: Ajit Khaparde @ 2009-08-18 11:25 UTC (permalink / raw)
To: davem, jgarzik, netdev
This patch adds support to flash a firmware image to a device using ethtool.
The driver gets the filename of the firmware image and flashes the image
using the request firmware path.
The region to be flashed - like redboot, phy will also be passed to the driver.
More options for other flash regions can be added in future.
The default behavior is to flash all the regions on the chip.
Signed-off-by: Ajit Khaparde <ajitk@serverengines.com>
---
include/linux/ethtool.h | 15 +++++++++++++++
net/core/ethtool.c | 17 +++++++++++++++++
2 files changed, 32 insertions(+), 0 deletions(-)
diff --git a/include/linux/ethtool.h b/include/linux/ethtool.h
index 90c4a36..7a5800e 100644
--- a/include/linux/ethtool.h
+++ b/include/linux/ethtool.h
@@ -362,6 +362,19 @@ struct ethtool_rxnfc {
__u32 rule_locs[0];
};
+enum ethtool_flash_op_type {
+ ETHTOOL_FLASH_ALL = 0,
+ ETHTOOL_FLASH_PHY,
+ ETHTOOL_FLASH_REDBOOT,
+};
+
+/* for passing firmware flashing related parameters */
+struct ethtool_flash {
+ __u32 cmd;
+ __u32 op_type;
+ char data[256];
+};
+
#ifdef __KERNEL__
struct net_device;
@@ -489,6 +502,7 @@ struct ethtool_ops {
int (*get_stats_count)(struct net_device *);/* use get_sset_count */
int (*get_rxnfc)(struct net_device *, struct ethtool_rxnfc *, void *);
int (*set_rxnfc)(struct net_device *, struct ethtool_rxnfc *);
+ int (*flash_device)(struct net_device *, u8 *, u32);
};
#endif /* __KERNEL__ */
@@ -545,6 +559,7 @@ struct ethtool_ops {
#define ETHTOOL_GRXCLSRLALL 0x00000030 /* Get all RX classification rule */
#define ETHTOOL_SRXCLSRLDEL 0x00000031 /* Delete RX classification rule */
#define ETHTOOL_SRXCLSRLINS 0x00000032 /* Insert RX classification rule */
+#define ETHTOOL_FLASHDEV 0x00000033 /* Flash firmware to device */
/* compatibility with older code */
#define SPARC_ETH_GSET ETHTOOL_GSET
diff --git a/net/core/ethtool.c b/net/core/ethtool.c
index 44e5711..fbdaade 100644
--- a/net/core/ethtool.c
+++ b/net/core/ethtool.c
@@ -898,6 +898,20 @@ static int ethtool_set_value(struct net_device *dev, char __user *useraddr,
return actor(dev, edata.data);
}
+static int ethtool_flash_device(struct net_device *dev, char __user *useraddr)
+{
+ int err;
+ struct ethtool_flash efl;
+
+ if (copy_from_user(&efl, useraddr, sizeof(efl)))
+ return -EFAULT;
+
+ if (!dev->ethtool_ops->flash_device)
+ return -EOPNOTSUPP;
+
+ return dev->ethtool_ops->flash_device(dev, &efl.data, efl.op_type);
+}
+
/* The main entry point in this file. Called from net/core/dev.c */
int dev_ethtool(struct net *net, struct ifreq *ifr)
@@ -1111,6 +1125,9 @@ int dev_ethtool(struct net *net, struct ifreq *ifr)
case ETHTOOL_SGRO:
rc = ethtool_set_gro(dev, useraddr);
break;
+ case ETHTOOL_FLASHDEV:
+ rc = ethtool_flash_device(dev, useraddr);
+ break;
default:
rc = -EOPNOTSUPP;
}
--
1.6.0.4
^ permalink raw reply related
* [PATCH] ethtool: Add "-f" option to flash a firmware image from the specified file to a device.
From: Ajit Khaparde @ 2009-08-18 11:25 UTC (permalink / raw)
To: davem, jgarzik, netdev
This patch adds a new "-f" option to the ethtool utility
to flash a firmware image specified by a file, to a network device.
The filename is passed to the network driver which will flash the image
on the chip using the request_firmware path.
The region to be flashed - like redboot, phy can be specified as an argument,
which will be passed to the driver.
More options for other flash regions can be added in future.
The default behavior is to flash all the regions on the chip.
Usage:
ethtool -f <interface name> <filename of firmware image>
ethtool -f <interface name> <filename of firmware image> [ all|redboot|phy ]
Signed-off-by: Ajit Khaparde <ajitk@serverengines.com>
---
ethtool-copy.h | 16 ++++++++++++
ethtool.c | 71 +++++++++++++++++++++++++++++++++++++++++++++++++++++++-
2 files changed, 86 insertions(+), 1 deletions(-)
diff --git a/ethtool-copy.h b/ethtool-copy.h
index 3ca4e2c..a7d11fb 100644
--- a/ethtool-copy.h
+++ b/ethtool-copy.h
@@ -272,6 +272,21 @@ struct ethtool_perm_addr {
__u8 data[0];
};
+#define ETHTOOL_FLASH_MAX_FILENAME 128
+#define ETHTOOL_FLASH_OP_TYPE_SIZE 32
+enum ethtool_flash_op_type {
+ ETHTOOL_FLASH_ALL = 0,
+ ETHTOOL_FLASH_PHY,
+ ETHTOOL_FLASH_REDBOOT,
+};
+
+/* for passing firmware flashing related parameters */
+struct ethtool_flash {
+ __u32 cmd;
+ __u32 op_type;
+ char data[256];
+};
+
/* boolean flags controlling per-interface behavior characteristics.
* When reading, the flag indicates whether or not a certain behavior
* is enabled/present. When writing, the flag indicates whether
@@ -338,6 +353,7 @@ struct ethtool_rxnfc {
#define ETHTOOL_SRXFH 0x0000002a /* Set RX flow hash configuration */
#define ETHTOOL_GGRO 0x0000002b /* Get GRO enable (ethtool_value) */
#define ETHTOOL_SGRO 0x0000002c /* Set GRO enable (ethtool_value) */
+#define ETHTOOL_FLASHDEV 0x00000033 /* Flash firmware to device */
/* compatibility with older code */
#define SPARC_ETH_GSET ETHTOOL_GSET
diff --git a/ethtool.c b/ethtool.c
index 0110682..d73ddc4 100644
--- a/ethtool.c
+++ b/ethtool.c
@@ -77,6 +77,7 @@ static char *unparse_rxfhashopts(u64 opts);
static int dump_rxfhash(int fhash, u64 val);
static int do_srxclass(int fd, struct ifreq *ifr);
static int do_grxclass(int fd, struct ifreq *ifr);
+static int do_flash(int fd, struct ifreq *ifr);
static int send_ioctl(int fd, struct ifreq *ifr);
static enum {
@@ -101,6 +102,7 @@ static enum {
MODE_GSTATS,
MODE_GNFC,
MODE_SNFC,
+ MODE_FLASHDEV,
} mode = MODE_GSET;
static struct option {
@@ -192,6 +194,9 @@ static struct option {
"classification options",
" [ rx-flow-hash tcp4|udp4|ah4|sctp4|"
"tcp6|udp6|ah6|sctp6 p|m|v|t|s|d|f|n|r... ]\n" },
+ { "-f", "--flash", MODE_FLASHDEV, "FILENAME " "Flash firmware image "
+ "from the specified file to device",
+ " [ all|redboot|phy ]\n" },
{ "-h", "--help", MODE_HELP, "Show this help" },
{}
};
@@ -304,6 +309,9 @@ static int rx_fhash_get = 0;
static int rx_fhash_set = 0;
static u32 rx_fhash_val = 0;
static int rx_fhash_changed = 0;
+static char flash_file[ETHTOOL_FLASH_MAX_FILENAME];
+static int flash = -1;
+static int flash_op = -1;
static enum {
ONLINE=0,
OFFLINE,
@@ -496,7 +504,8 @@ static void parse_cmdline(int argc, char **argp)
(mode == MODE_GSTATS) ||
(mode == MODE_GNFC) ||
(mode == MODE_SNFC) ||
- (mode == MODE_PHYS_ID)) {
+ (mode == MODE_PHYS_ID) ||
+ (mode == MODE_FLASHDEV)) {
devname = argp[i];
break;
}
@@ -516,6 +525,10 @@ static void parse_cmdline(int argc, char **argp)
if (phys_id_time < 0)
show_usage(1);
break;
+ } else if (mode == MODE_FLASHDEV) {
+ sprintf(flash_file, "%s", argp[i]);
+ flash = 1;
+ break;
}
/* fallthrough */
default:
@@ -590,6 +603,17 @@ static void parse_cmdline(int argc, char **argp)
show_usage(1);
break;
}
+ if (mode == MODE_FLASHDEV) {
+ if (!strcmp(argp[i], "redboot"))
+ flash_op = ETHTOOL_FLASH_REDBOOT;
+ else if (!strcmp(argp[i], "phy"))
+ flash_op = ETHTOOL_FLASH_PHY;
+ else if (!strcmp(argp[i], "all"))
+ flash_op = ETHTOOL_FLASH_ALL;
+ else
+ show_usage(1);
+ break;
+ }
if (mode == MODE_SNFC) {
if (!strcmp(argp[i], "rx-flow-hash")) {
i += 1;
@@ -1504,6 +1528,8 @@ static int doit(void)
return do_grxclass(fd, &ifr);
} else if (mode == MODE_SNFC) {
return do_srxclass(fd, &ifr);
+ } else if (mode == MODE_FLASHDEV) {
+ return do_flash(fd, &ifr);
}
return 69;
@@ -2398,6 +2424,49 @@ static int do_grxclass(int fd, struct ifreq *ifr)
return 0;
}
+static int do_flash(int fd, struct ifreq *ifr)
+{
+ struct ethtool_flash efl;
+ int err;
+ char path[ETHTOOL_FLASH_MAX_FILENAME * 2] = "/lib/firmware/";
+ FILE *f;
+
+ if (flash < 0) {
+ fprintf(stdout, "Missing filename argument\n");
+ show_usage(1);
+ return -EPERM;
+ }
+
+ strcat(path, flash_file);
+
+ if (strlen(flash_file) > ETHTOOL_FLASH_MAX_FILENAME) {
+ fprintf(stdout, "Filename too long\n");
+ return -EINVAL;
+ }
+
+ f = fopen(path, "r");
+ if (!f) {
+ perror(path);
+ return -ENOENT;
+ }
+ fclose(f);
+
+ efl.cmd = ETHTOOL_FLASHDEV;
+ sprintf(efl.data, "%s", flash_file);
+
+ if (flash_op < 0)
+ efl.op_type = ETHTOOL_FLASH_ALL;
+ else
+ efl.op_type = flash_op;
+
+ ifr->ifr_data = (caddr_t)&efl;
+ err = send_ioctl(fd, ifr);
+ if (err < 0)
+ perror("Flashing failed");
+
+ return err;
+}
+
static int send_ioctl(int fd, struct ifreq *ifr)
{
return ioctl(fd, SIOCETHTOOL, ifr);
--
1.6.0.4
^ permalink raw reply related
* Re: [PATCH v3 3/6] vbus: add a "vbus-proxy" bus model for vbus_driver objects
From: Avi Kivity @ 2009-08-18 11:15 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Gregory Haskins, Ingo Molnar, Gregory Haskins, kvm,
alacrityvm-devel, linux-kernel, netdev
In-Reply-To: <20090818110708.GG13878@redhat.com>
On 08/18/2009 02:07 PM, Michael S. Tsirkin wrote:
> On Tue, Aug 18, 2009 at 01:45:05PM +0300, Avi Kivity wrote:
>
>> On 08/18/2009 01:28 PM, Michael S. Tsirkin wrote:
>>
>>>
>>>> Suppose a nested guest has two devices. One a virtual device backed by
>>>> its host (our guest), and one a virtual device backed by us (the real
>>>> host), and assigned by the guest to the nested guest. If both devices
>>>> use hypercalls, there is no way to distinguish between them.
>>>>
>>>>
>>> Not sure I understand. What I had in mind is that devices would have to
>>> either use different hypercalls and map hypercall to address during
>>> setup, or pass address with each hypercall. We get the hypercall,
>>> translate the address as if it was pio access, and know the destination?
>>>
>>>
>> There are no different hypercalls. There's just one hypercall
>> instruction, and there's no standard on how it's used. If a nested call
>> issues a hypercall instruction, you have no idea if it's calling a
>> Hyper-V hypercall or a vbus/virtio kick.
>>
> userspace will know which it is, because hypercall capability
> in the device has been activated, and can tell kernel, using
> something similar to iosignalfd. No?
>
The host kernel sees a hypercall vmexit. How does it know if it's a
nested-guest-to-guest hypercall or a nested-guest-to-host hypercall?
The two are equally valid at the same time.
>> You could have a protocol where you register the hypercall instruction's
>> address with its recipient, but it quickly becomes a tangled mess.
>>
> I really thought we could pass the io address in register as an input
> parameter. Is there a way to do this in a secure manner?
>
> Hmm. Doesn't kvm use hypercalls now? How does this work with nesting?
> For example, in this code in arch/x86/kvm/x86.c:
>
> switch (nr) {
> case KVM_HC_VAPIC_POLL_IRQ:
> ret = 0;
> break;
> case KVM_HC_MMU_OP:
> r = kvm_pv_mmu_op(vcpu, a0, hc_gpa(vcpu, a1, a2),&ret);
> break;
> default:
> ret = -KVM_ENOSYS;
> break;
> }
>
> how do we know that it's the guest and not the nested guest performing
> the hypercall?
>
The host knows whether the guest or nested guest are running. If the
guest is running, it's a guest-to-host hypercall. If the nested guest
is running, it's a nested-guest-to-guest hypercall. We don't have
nested-guest-to-host hypercalls (and couldn't unless we get agreement on
a protocol from all hypervisor vendors).
--
error compiling committee.c: too many arguments to function
^ permalink raw reply
* Re: [PATCH v3 3/6] vbus: add a "vbus-proxy" bus model for vbus_driver objects
From: Michael S. Tsirkin @ 2009-08-18 11:07 UTC (permalink / raw)
To: Avi Kivity
Cc: Gregory Haskins, Ingo Molnar, Gregory Haskins, kvm,
alacrityvm-devel, linux-kernel, netdev
In-Reply-To: <4A8A8631.1040006@redhat.com>
On Tue, Aug 18, 2009 at 01:45:05PM +0300, Avi Kivity wrote:
> On 08/18/2009 01:28 PM, Michael S. Tsirkin wrote:
>>
>>> Suppose a nested guest has two devices. One a virtual device backed by
>>> its host (our guest), and one a virtual device backed by us (the real
>>> host), and assigned by the guest to the nested guest. If both devices
>>> use hypercalls, there is no way to distinguish between them.
>>>
>> Not sure I understand. What I had in mind is that devices would have to
>> either use different hypercalls and map hypercall to address during
>> setup, or pass address with each hypercall. We get the hypercall,
>> translate the address as if it was pio access, and know the destination?
>>
>
> There are no different hypercalls. There's just one hypercall
> instruction, and there's no standard on how it's used. If a nested call
> issues a hypercall instruction, you have no idea if it's calling a
> Hyper-V hypercall or a vbus/virtio kick.
userspace will know which it is, because hypercall capability
in the device has been activated, and can tell kernel, using
something similar to iosignalfd. No?
> You could have a protocol where you register the hypercall instruction's
> address with its recipient, but it quickly becomes a tangled mess.
I really thought we could pass the io address in register as an input
parameter. Is there a way to do this in a secure manner?
Hmm. Doesn't kvm use hypercalls now? How does this work with nesting?
For example, in this code in arch/x86/kvm/x86.c:
switch (nr) {
case KVM_HC_VAPIC_POLL_IRQ:
ret = 0;
break;
case KVM_HC_MMU_OP:
r = kvm_pv_mmu_op(vcpu, a0, hc_gpa(vcpu, a1, a2), &ret);
break;
default:
ret = -KVM_ENOSYS;
break;
}
how do we know that it's the guest and not the nested guest performing
the hypercall?
> And for what? pio and hypercalls have the same performance characteristics.
No idea about that. I'm assuming Gregory knows why he wants to use hypercalls,
I was just trying to help find a way that is also palatable, and flexible.
> --
> error compiling committee.c: too many arguments to function
^ permalink raw reply
* Re: [PATCH v3 3/6] vbus: add a "vbus-proxy" bus model for vbus_driver objects
From: Avi Kivity @ 2009-08-18 10:45 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Gregory Haskins, Ingo Molnar, Gregory Haskins, kvm,
alacrityvm-devel, linux-kernel, netdev
In-Reply-To: <20090818102840.GF13878@redhat.com>
On 08/18/2009 01:28 PM, Michael S. Tsirkin wrote:
>
>> Suppose a nested guest has two devices. One a virtual device backed by
>> its host (our guest), and one a virtual device backed by us (the real
>> host), and assigned by the guest to the nested guest. If both devices
>> use hypercalls, there is no way to distinguish between them.
>>
> Not sure I understand. What I had in mind is that devices would have to
> either use different hypercalls and map hypercall to address during
> setup, or pass address with each hypercall. We get the hypercall,
> translate the address as if it was pio access, and know the destination?
>
There are no different hypercalls. There's just one hypercall
instruction, and there's no standard on how it's used. If a nested call
issues a hypercall instruction, you have no idea if it's calling a
Hyper-V hypercall or a vbus/virtio kick.
You could have a protocol where you register the hypercall instruction's
address with its recipient, but it quickly becomes a tangled mess. And
for what? pio and hypercalls have the same performance characteristics.
--
error compiling committee.c: too many arguments to function
^ permalink raw reply
* Re: bad nat connection tracking performance with ip_gre
From: Patrick McHardy @ 2009-08-18 10:38 UTC (permalink / raw)
To: Timo Teräs; +Cc: netfilter-devel, netdev
In-Reply-To: <4A8A7F14.3010103@iki.fi>
Timo Teräs wrote:
> Hi,
>
> I noticed (in relation to my nbma gre multicast testing) that
> the nat connection tracking code does not cache flows for
> locally originating traffic that is routed to gre tunnel
> (forwarded traffic is ok).
>
> I basically have a router box with nbma gre tunnel. It gets
> 10/8 traffic. And is routed to internet interface. An ipsec
> xfrm is applied.
>
> Now, if the router box is forwarding traffic from some
> physical interface, everything works as expected.
>
> However, if a local process on the router box is sending
> packets that go to gre tunnel, each packet causes a new
> lookup on nat table OUTPUT chain. This is easily verified
> by doing flood ping on router box on private IP and the
> counters on nat table OUTPUT chain default policy start
> to get incremented wildly.
>
> I tried to oprofile this and it says most of the time is
> spent in ipt_do_table(). I would suppose that the place
> where netfilter hook is called is
> ip_gre.c:ipgre_tunnel_xmit() when it invokes macro
> IPTUNNEL_XMIT() calling ip_local_out().
>
> Monitoring the connection tracking stats, it looks like
> all packets are reusing the proper connection tracking
> cache entry. But somehow the nat target still gets
> called for the locally originating packets to gre.
>
> Any ideas how to fix this?
Please use the TRACE target in raw/OUTPUT to trace the flow of
packets through the netfilter hooks:
modprobe ipt_LOG
iptables -t raw -A OUTPUT -j TRACE
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH v3 3/6] vbus: add a "vbus-proxy" bus model for vbus_driver objects
From: Michael S. Tsirkin @ 2009-08-18 10:28 UTC (permalink / raw)
To: Avi Kivity
Cc: Gregory Haskins, Ingo Molnar, Gregory Haskins, kvm,
alacrityvm-devel, linux-kernel, netdev
In-Reply-To: <4A8A7EE5.6090209@redhat.com>
On Tue, Aug 18, 2009 at 01:13:57PM +0300, Avi Kivity wrote:
> On 08/18/2009 01:09 PM, Michael S. Tsirkin wrote:
>>
>>> mmio and pio don't have this problem since the host can use the address
>>> to locate the destination.
>>>
>> So userspace could map hypercall to address during setup and tell the
>> host kernel?
>>
>
> Suppose a nested guest has two devices. One a virtual device backed by
> its host (our guest), and one a virtual device backed by us (the real
> host), and assigned by the guest to the nested guest. If both devices
> use hypercalls, there is no way to distinguish between them.
Not sure I understand. What I had in mind is that devices would have to
either use different hypercalls and map hypercall to address during
setup, or pass address with each hypercall. We get the hypercall,
translate the address as if it was pio access, and know the destination?
> --
> error compiling committee.c: too many arguments to function
^ permalink raw reply
* bad nat connection tracking performance with ip_gre
From: Timo Teräs @ 2009-08-18 10:14 UTC (permalink / raw)
To: netfilter-devel; +Cc: netdev
Hi,
I noticed (in relation to my nbma gre multicast testing) that
the nat connection tracking code does not cache flows for
locally originating traffic that is routed to gre tunnel
(forwarded traffic is ok).
I basically have a router box with nbma gre tunnel. It gets
10/8 traffic. And is routed to internet interface. An ipsec
xfrm is applied.
Now, if the router box is forwarding traffic from some
physical interface, everything works as expected.
However, if a local process on the router box is sending
packets that go to gre tunnel, each packet causes a new
lookup on nat table OUTPUT chain. This is easily verified
by doing flood ping on router box on private IP and the
counters on nat table OUTPUT chain default policy start
to get incremented wildly.
I tried to oprofile this and it says most of the time is
spent in ipt_do_table(). I would suppose that the place
where netfilter hook is called is
ip_gre.c:ipgre_tunnel_xmit() when it invokes macro
IPTUNNEL_XMIT() calling ip_local_out().
Monitoring the connection tracking stats, it looks like
all packets are reusing the proper connection tracking
cache entry. But somehow the nat target still gets
called for the locally originating packets to gre.
Any ideas how to fix this?
Thanks,
Timo
^ permalink raw reply
* Re: [PATCH v3 3/6] vbus: add a "vbus-proxy" bus model for vbus_driver objects
From: Avi Kivity @ 2009-08-18 10:13 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Gregory Haskins, Ingo Molnar, Gregory Haskins, kvm,
alacrityvm-devel, linux-kernel, netdev
In-Reply-To: <20090818100945.GD13878@redhat.com>
On 08/18/2009 01:09 PM, Michael S. Tsirkin wrote:
>
>> mmio and pio don't have this problem since the host can use the address
>> to locate the destination.
>>
> So userspace could map hypercall to address during setup and tell the
> host kernel?
>
Suppose a nested guest has two devices. One a virtual device backed by
its host (our guest), and one a virtual device backed by us (the real
host), and assigned by the guest to the nested guest. If both devices
use hypercalls, there is no way to distinguish between them.
--
error compiling committee.c: too many arguments to function
^ permalink raw reply
* Re: [PATCH v3 3/6] vbus: add a "vbus-proxy" bus model for vbus_driver objects
From: Michael S. Tsirkin @ 2009-08-18 10:09 UTC (permalink / raw)
To: Avi Kivity
Cc: Gregory Haskins, Ingo Molnar, Gregory Haskins, kvm,
alacrityvm-devel, linux-kernel, netdev
In-Reply-To: <4A8A7BB9.2020906@redhat.com>
On Tue, Aug 18, 2009 at 01:00:25PM +0300, Avi Kivity wrote:
> On 08/18/2009 12:53 PM, Michael S. Tsirkin wrote:
>> I'm not hung up on PCI, myself. An idea that might help you get Avi
>> on-board: do setup in userspace, over PCI. Negotiate hypercall support
>> (e.g. with a PCI capability) and then switch to that for fastpath. Hmm?
>>
>
> Hypercalls don't nest well. When a nested guest issues a hypercall, you
> have to assume it is destined to the enclosing guest, so you can't
> assign a hypercall-capable device to a nested guest.
>
> mmio and pio don't have this problem since the host can use the address
> to locate the destination.
So userspace could map hypercall to address during setup and tell the
host kernel?
> --
> error compiling committee.c: too many arguments to function
^ permalink raw reply
* Re: [PATCH v3 3/6] vbus: add a "vbus-proxy" bus model for vbus_driver objects
From: Avi Kivity @ 2009-08-18 10:00 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Gregory Haskins, Ingo Molnar, Gregory Haskins, kvm,
alacrityvm-devel, linux-kernel, netdev
In-Reply-To: <20090818095313.GC13878@redhat.com>
On 08/18/2009 12:53 PM, Michael S. Tsirkin wrote:
> I'm not hung up on PCI, myself. An idea that might help you get Avi
> on-board: do setup in userspace, over PCI. Negotiate hypercall support
> (e.g. with a PCI capability) and then switch to that for fastpath. Hmm?
>
Hypercalls don't nest well. When a nested guest issues a hypercall, you
have to assume it is destined to the enclosing guest, so you can't
assign a hypercall-capable device to a nested guest.
mmio and pio don't have this problem since the host can use the address
to locate the destination.
--
error compiling committee.c: too many arguments to function
^ permalink raw reply
* Re: [PATCH v3 3/6] vbus: add a "vbus-proxy" bus model for vbus_driver objects
From: Michael S. Tsirkin @ 2009-08-18 9:53 UTC (permalink / raw)
To: Gregory Haskins
Cc: Ingo Molnar, Gregory Haskins, kvm, Avi Kivity, alacrityvm-devel,
linux-kernel, netdev
In-Reply-To: <4A89B08A.4010103@gmail.com>
On Mon, Aug 17, 2009 at 03:33:30PM -0400, Gregory Haskins wrote:
> There is a secondary question of venet (a vbus native device) verses
> virtio-net (a virtio native device that works with PCI or VBUS). If
> this contention is really around venet vs virtio-net, I may possibly
> conceed and retract its submission to mainline.
For me yes, venet+ioq competing with virtio+virtqueue.
> I've been pushing it to date because people are using it and I don't
> see any reason that the driver couldn't be upstream.
If virtio is just as fast, they can just use it without knowing it.
Clearly, that's better since we support virtio anyway ...
> -- Issues --
>
> Out of all this, I think the biggest contention point is the design of
> the vbus-connector that I use in AlacrityVM (Avi, correct me if I am
> wrong and you object to other aspects as well). I suspect that if I had
> designed the vbus-connector to surface vbus devices as PCI devices via
> QEMU, the patches would potentially have been pulled in a while ago.
>
> There are, of course, reasons why vbus does *not* render as PCI, so this
> is the meat of of your question, I believe.
>
> At a high level, PCI was designed for software-to-hardware interaction,
> so it makes assumptions about that relationship that do not necessarily
> apply to virtualization.
I'm not hung up on PCI, myself. An idea that might help you get Avi
on-board: do setup in userspace, over PCI. Negotiate hypercall support
(e.g. with a PCI capability) and then switch to that for fastpath. Hmm?
> As another example, the connector design coalesces *all* shm-signals
> into a single interrupt (by prio) that uses the same context-switch
> mitigation techniques that help boost things like networking. This
> effectively means we can detect and optimize out ack/eoi cycles from the
> APIC as the IO load increases (which is when you need it most). PCI has
> no such concept.
Could you elaborate on this one for me? How does context-switch
mitigation work?
> In addition, the signals and interrupts are priority aware, which is
> useful for things like 802.1p networking where you may establish 8-tx
> and 8-rx queues for your virtio-net device. x86 APIC really has no
> usable equivalent, so PCI is stuck here.
By the way, multiqueue support in virtio would be very nice to have,
and seems mostly unrelated to vbus.
--
MST
^ permalink raw reply
* Re: [PATCH v3 3/6] vbus: add a "vbus-proxy" bus model for vbus_driver objects
From: Michael S. Tsirkin @ 2009-08-18 8:54 UTC (permalink / raw)
To: Anthony Liguori
Cc: Gregory Haskins, Ingo Molnar, Gregory Haskins, kvm, Avi Kivity,
alacrityvm-devel, linux-kernel, netdev
In-Reply-To: <4A89FF08.30509@codemonkey.ws>
On Mon, Aug 17, 2009 at 08:08:24PM -0500, Anthony Liguori wrote:
> In particular, I think it would be a big win to avoid knowledge of slots in
> the kernel by doing ring translation in userspace.
vhost supports this BTW: just don't call the memory table ioctl.
In-kernel translation is simple, as well, though.
--
MST
^ permalink raw reply
* Re: [PATCH] yellowfin: Fix buffer underrun after dev_alloc_skb() failure
From: Roel Kluin @ 2009-08-18 8:50 UTC (permalink / raw)
To: David Miller; +Cc: netdev, akpm
In-Reply-To: <20090814.161701.02803885.davem@davemloft.net>
yellowfin_init_ring() needs to clean up if dev_alloc_skb() fails and
should pass an error status up to the caller. This also prevents an
buffer underrun if failure occurred in the first iteration.
yellowfin_open() which calls yellowfin_init_ring() should free its
requested irq upon failure.
Signed-off-by: Roel Kluin <roel.kluin@gmail.com>
---
diff --git a/drivers/net/yellowfin.c b/drivers/net/yellowfin.c
index a075801..c2fd618 100644
--- a/drivers/net/yellowfin.c
+++ b/drivers/net/yellowfin.c
@@ -346,7 +346,7 @@ static int netdev_ioctl(struct net_device *dev, struct ifreq *rq, int cmd);
static int yellowfin_open(struct net_device *dev);
static void yellowfin_timer(unsigned long data);
static void yellowfin_tx_timeout(struct net_device *dev);
-static void yellowfin_init_ring(struct net_device *dev);
+static int yellowfin_init_ring(struct net_device *dev);
static int yellowfin_start_xmit(struct sk_buff *skb, struct net_device *dev);
static irqreturn_t yellowfin_interrupt(int irq, void *dev_instance);
static int yellowfin_rx(struct net_device *dev);
@@ -573,19 +573,24 @@ static int yellowfin_open(struct net_device *dev)
{
struct yellowfin_private *yp = netdev_priv(dev);
void __iomem *ioaddr = yp->base;
- int i;
+ int i, ret;
/* Reset the chip. */
iowrite32(0x80000000, ioaddr + DMACtrl);
- i = request_irq(dev->irq, &yellowfin_interrupt, IRQF_SHARED, dev->name, dev);
- if (i) return i;
+ ret = request_irq(dev->irq, &yellowfin_interrupt, IRQF_SHARED, dev->name, dev);
+ if (ret)
+ return ret;
if (yellowfin_debug > 1)
printk(KERN_DEBUG "%s: yellowfin_open() irq %d.\n",
dev->name, dev->irq);
- yellowfin_init_ring(dev);
+ ret = yellowfin_init_ring(dev);
+ if (ret) {
+ free_irq(dev->irq, dev);
+ return ret;
+ }
iowrite32(yp->rx_ring_dma, ioaddr + RxPtr);
iowrite32(yp->tx_ring_dma, ioaddr + TxPtr);
@@ -725,10 +730,10 @@ static void yellowfin_tx_timeout(struct net_device *dev)
}
/* Initialize the Rx and Tx rings, along with various 'dev' bits. */
-static void yellowfin_init_ring(struct net_device *dev)
+static int yellowfin_init_ring(struct net_device *dev)
{
struct yellowfin_private *yp = netdev_priv(dev);
- int i;
+ int i, j;
yp->tx_full = 0;
yp->cur_rx = yp->cur_tx = 0;
@@ -753,6 +758,11 @@ static void yellowfin_init_ring(struct net_device *dev)
yp->rx_ring[i].addr = cpu_to_le32(pci_map_single(yp->pci_dev,
skb->data, yp->rx_buf_sz, PCI_DMA_FROMDEVICE));
}
+ if (i != RX_RING_SIZE) {
+ for (j = 0; j < i; j++)
+ dev_kfree_skb(yp->rx_skbuff[j]);
+ return -ENOMEM;
+ }
yp->rx_ring[i-1].dbdma_cmd = cpu_to_le32(CMD_STOP);
yp->dirty_rx = (unsigned int)(i - RX_RING_SIZE);
@@ -769,8 +779,6 @@ static void yellowfin_init_ring(struct net_device *dev)
yp->tx_ring[--i].dbdma_cmd = cpu_to_le32(CMD_STOP | BRANCH_ALWAYS);
#else
{
- int j;
-
/* Tx ring needs a pair of descriptors, the second for the status. */
for (i = 0; i < TX_RING_SIZE; i++) {
j = 2*i;
@@ -805,7 +813,7 @@ static void yellowfin_init_ring(struct net_device *dev)
}
#endif
yp->tx_tail_desc = &yp->tx_status[0];
- return;
+ return 0;
}
static int yellowfin_start_xmit(struct sk_buff *skb, struct net_device *dev)
^ permalink raw reply related
* Re: [PATCH v3 3/6] vbus: add a "vbus-proxy" bus model for vbus_driver objects
From: Michael S. Tsirkin @ 2009-08-18 8:46 UTC (permalink / raw)
To: Gregory Haskins
Cc: Anthony Liguori, Ingo Molnar, Gregory Haskins, kvm, Avi Kivity,
alacrityvm-devel, linux-kernel, netdev
In-Reply-To: <4A89BAC5.9040400@gmail.com>
On Mon, Aug 17, 2009 at 04:17:09PM -0400, Gregory Haskins wrote:
> Michael S. Tsirkin wrote:
> > On Mon, Aug 17, 2009 at 10:14:56AM -0400, Gregory Haskins wrote:
> >> Case in point: Take an upstream kernel and you can modprobe the
> >> vbus-pcibridge in and virtio devices will work over that transport
> >> unmodified.
> >>
> >> See http://lkml.org/lkml/2009/8/6/244 for details.
> >
> > The modprobe you are talking about would need
> > to be done in guest kernel, correct?
>
> Yes, and your point is? "unmodified" (pardon the psuedo pun) modifies
> "virtio", not "guest".
> It means you can take an off-the-shelf kernel
> with off-the-shelf virtio (ala distro-kernel) and modprobe
> vbus-pcibridge and get alacrityvm acceleration.
Heh, by that logic ksplice does not modify running kernel either :)
> It is not a design goal of mine to forbid the loading of a new driver,
> so I am ok with that requirement.
>
> >> OTOH, Michael's patch is purely targeted at improving virtio-net on kvm,
> >> and its likewise constrained by various limitations of that decision
> >> (such as its reliance of the PCI model, and the kvm memory scheme).
> >
> > vhost is actually not related to PCI in any way. It simply leaves all
> > setup for userspace to do. And the memory scheme was intentionally
> > separated from kvm so that it can easily support e.g. lguest.
> >
>
> I think you have missed my point. I mean that vhost requires a separate
> bus-model (ala qemu-pci).
So? That can be in userspace, and can be anything including vbus.
> And no, your memory scheme is not separated,
> at least, not very well. It still assumes memory-regions and
> copy_to_user(), which is very kvm-esque.
I don't think so: works for lguest, kvm, UML and containers
> Vbus has people using things
> like userspace containers (no regions),
vhost by default works without regions
> and physical hardware (dma
> controllers, so no regions or copy_to_user) so your scheme quickly falls
> apart once you get away from KVM.
Someone took a driver and is building hardware for it ... so what?
> Don't get me wrong: That design may have its place. Perhaps you only
> care about fixing KVM, which is a perfectly acceptable strategy.
> Its just not a strategy that I think is the best approach. Essentially you
> are promoting the proliferation of competing backends, and I am trying
> to unify them (which is ironic that this thread started with concerns I
> was fragmenting things ;).
So, you don't see how venet fragments things? It's pretty obvious ...
> The bottom line is, you have a simpler solution that is more finely
> targeted at KVM and virtio-networking. It fixes probably a lot of
> problems with the existing implementation, but it still has limitations.
>
> OTOH, what I am promoting is more complex, but more flexible. That is
> the tradeoff. You can't have both ;)
We can. connect eventfds to hypercalls, and vhost will work with vbus.
> So do not for one second think
> that what you implemented is equivalent, because they are not.
>
> In fact, I believe I warned you about this potential problem when you
> decided to implement your own version. I think I said something to the
> effect of "you will either have a subset of functionality, or you will
> ultimately reinvent what I did". Right now you are in the subset phase.
No. Unlike vbus, vhost supports unmodified guests and live migration.
> Perhaps someday you will be in the complete-reinvent phase. Why you
> wanted to go that route when I had already worked though the issues is
> something perhaps only you will ever know, but I'm sure you had your
> reasons. But do note you could have saved yourself grief by reusing my
> already implemented and tested variant, as I politely offered to work
> with you on making it meet your needs.
> Kind Regards
> -Greg
>
you have a midlayer. I could not use it without pulling in all of it.
--
MST
^ permalink raw reply
* Re: [PATCH v3 3/6] vbus: add a "vbus-proxy" bus model for vbus_driver objects
From: Avi Kivity @ 2009-08-18 8:33 UTC (permalink / raw)
To: Gregory Haskins
Cc: Ingo Molnar, Gregory Haskins, kvm, alacrityvm-devel, linux-kernel,
netdev, Michael S. Tsirkin
In-Reply-To: <4A89B08A.4010103@gmail.com>
On 08/17/2009 10:33 PM, Gregory Haskins wrote:
>
> There is a secondary question of venet (a vbus native device) verses
> virtio-net (a virtio native device that works with PCI or VBUS). If
> this contention is really around venet vs virtio-net, I may possibly
> conceed and retract its submission to mainline. I've been pushing it to
> date because people are using it and I don't see any reason that the
> driver couldn't be upstream.
>
That's probably the cause of much confusion. The primary kvm pain point
is now networking, so in any vbus discussion we're concentrating on that
aspect.
>> Also, are you willing to help virtio to become faster?
>>
> Yes, that is not a problem. Note that virtio in general, and
> virtio-net/venet in particular are not the primary goal here, however.
> Improved 802.x and block IO are just positive side-effects of the
> effort. I started with 802.x networking just to demonstrate the IO
> layer capabilities, and to test it. It ended up being so good on
> contrast to existing facilities, that developers in the vbus community
> started using it for production development.
>
> Ultimately, I created vbus to address areas of performance that have not
> yet been addressed in things like KVM. Areas such as real-time guests,
> or RDMA (host bypass) interfaces.
Can you explain how vbus achieves RDMA?
I also don't see the connection to real time guests.
> I also designed it in such a way that
> we could, in theory, write one set of (linux-based) backends, and have
> them work across a variety of environments (such as containers/VMs like
> KVM, lguest, openvz, but also physical systems like blade enclosures and
> clusters, or even applications running on the host).
>
Sorry, I'm still confused. Why would openvz need vbus? It already has
zero-copy networking since it's a shared kernel. Shared memory should
also work seamlessly, you just need to expose the shared memory object
on a shared part of the namespace. And of course, anything in the
kernel is already shared.
>> Or do you
>> have arguments why that is impossible to do so and why the only
>> possible solution is vbus? Avi says no such arguments were offered
>> so far.
>>
> Not for lack of trying. I think my points have just been missed
> everytime I try to describe them. ;) Basically I write a message very
> similar to this one, and the next conversation starts back from square
> one. But I digress, let me try again..
>
> Noting that this discussion is really about the layer *below* virtio,
> not virtio itself (e.g. PCI vs vbus). Lets start with a little background:
>
> -- Background --
>
> So on one level, we have the resource-container technology called
> "vbus". It lets you create a container on the host, fill it with
> virtual devices, and assign that container to some context (such as a
> KVM guest). These "devices" are LKMs, and each device has a very simple
> verb namespace consisting of a synchronous "call()" method, and a
> "shm()" method for establishing async channels.
>
> The async channels are just shared-memory with a signal path (e.g.
> interrupts and hypercalls), which the device+driver can use to overlay
> things like rings (virtqueues, IOQs), or other shared-memory based
> constructs of their choosing (such as a shared table). The signal path
> is designed to minimize enter/exits and reduce spurious signals in a
> unified way (see shm-signal patch).
>
> call() can be used both for config-space like details, as well as
> fast-path messaging that require synchronous behavior (such as guest
> scheduler updates).
>
> All of this is managed via sysfs/configfs.
>
One point of contention is that this is all managementy stuff and should
be kept out of the host kernel. Exposing shared memory, interrupts, and
guest hypercalls can all be easily done from userspace (as virtio
demonstrates). True, some devices need kernel acceleration, but that's
no reason to put everything into the host kernel.
> On the guest, we have a "vbus-proxy" which is how the guest gets access
> to devices assigned to its container. (as an aside, "virtio" devices
> can be populated in the container, and then surfaced up to the
> virtio-bus via that virtio-vbus patch I mentioned).
>
> There is a thing called a "vbus-connector" which is the guest specific
> part. Its job is to connect the vbus-proxy in the guest, to the vbus
> container on the host. How it does its job is specific to the connector
> implementation, but its role is to transport messages between the guest
> and the host (such as for call() and shm() invocations) and to handle
> things like discovery and hotswap.
>
virtio has an exact parallel here (virtio-pci and friends).
> Out of all this, I think the biggest contention point is the design of
> the vbus-connector that I use in AlacrityVM (Avi, correct me if I am
> wrong and you object to other aspects as well). I suspect that if I had
> designed the vbus-connector to surface vbus devices as PCI devices via
> QEMU, the patches would potentially have been pulled in a while ago.
>
Exposing devices as PCI is an important issue for me, as I have to
consider non-Linux guests.
Another issue is the host kernel management code which I believe is
superfluous.
But the biggest issue is compatibility. virtio exists and has Windows
and Linux drivers. Without a fatal flaw in virtio we'll continue to
support it. Given that, why spread to a new model?
Of course, I understand you're interested in non-ethernet, non-block
devices. I can't comment on these until I see them. Maybe they can fit
the virtio model, and maybe they can't.
> There are, of course, reasons why vbus does *not* render as PCI, so this
> is the meat of of your question, I believe.
>
> At a high level, PCI was designed for software-to-hardware interaction,
> so it makes assumptions about that relationship that do not necessarily
> apply to virtualization.
>
> For instance:
>
> A) hardware can only generate byte/word sized requests at a time because
> that is all the pcb-etch and silicon support. So hardware is usually
> expressed in terms of some number of "registers".
>
No, hardware happily DMAs to and fro main memory. Some hardware of
course uses mmio registers extensively, but not virtio hardware. With
the recent MSI support no registers are touched in the fast path.
> C) the target end-point has no visibility into the CPU machine state
> other than the parameters passed in the bus-cycle (usually an address
> and data tuple).
>
That's not an issue. Accessing memory is cheap.
> D) device-ids are in a fixed width register and centrally assigned from
> an authority (e.g. PCI-SIG).
>
That's not an issue either. Qumranet/Red Hat has donated a range of
device IDs for use in virtio. Device IDs are how devices are associated
with drivers, so you'll need something similar for vbus.
> E) Interrupt/MSI routing is per-device oriented
>
Please elaborate. What is the issue? How does vbus solve it?
> F) Interrupts/MSI are assumed cheap to inject
>
Interrupts are not assumed cheap; that's why interrupt mitigation is
used (on real and virtual hardware).
> G) Interrupts/MSI are non-priortizable.
>
They are prioritizable; Linux ignores this though (Windows doesn't).
Please elaborate on what the problem is and how vbus solves it.
> H) Interrupts/MSI are statically established
>
Can you give an example of why this is a problem?
> These assumptions and constraints may be completely different or simply
> invalid in a virtualized guest. For instance, the hypervisor is just
> software, and therefore it's not restricted to "etch" constraints. IO
> requests can be arbitrarily large, just as if you are invoking a library
> function-call or OS system-call. Likewise, each one of those requests is
> a branch and a context switch, so it has often has greater performance
> implications than a simple register bus-cycle in hardware. If you use
> an MMIO variant, it has to run through the page-fault code to be decoded.
>
> The result is typically decreased performance if you try to do the same
> thing real hardware does. This is why you usually see hypervisor
> specific drivers (e.g. virtio-net, vmnet, etc) a common feature.
>
> _Some_ performance oriented items can technically be accomplished in
> PCI, albeit in a much more awkward way. For instance, you can set up a
> really fast, low-latency "call()" mechanism using a PIO port on a
> PCI-model and ioeventfd. As a matter of fact, this is exactly what the
> vbus pci-bridge does:
>
What performance oriented items have been left unaddressed?
virtio and vbus use three communications channels: call from guest to
host (implemented as pio and reasonably fast), call from host to guest
(implemented as msi and reasonably fast) and shared memory (as fast as
it can be). Where does PCI limit you in any way?
> The problem here is that this is incredibly awkward to setup. You have
> all that per-cpu goo and the registration of the memory on the guest.
> And on the host side, you have all the vmapping of the registered
> memory, and the file-descriptor to manage. In short, its really painful.
>
> I would much prefer to do this *once*, and then let all my devices
> simple re-use that infrastructure. This is, in fact, what I do. Here
> is the device model that a guest sees:
>
virtio also reuses the pci code, on both guest and host.
> Moving on: _Other_ items cannot be replicated (at least, not without
> hacking it into something that is no longer PCI.
>
> Things like the pci-id namespace are just silly for software. I would
> rather have a namespace that does not require central management so
> people are free to create vbus-backends at will. This is akin to
> registering a device MAJOR/MINOR, verses using the various dynamic
> assignment mechanisms. vbus uses a string identifier in place of a
> pci-id. This is superior IMHO, and not compatible with PCI.
>
How do you handle conflicts? Again you need a central authority to hand
out names or prefixes.
> As another example, the connector design coalesces *all* shm-signals
> into a single interrupt (by prio) that uses the same context-switch
> mitigation techniques that help boost things like networking. This
> effectively means we can detect and optimize out ack/eoi cycles from the
> APIC as the IO load increases (which is when you need it most). PCI has
> no such concept.
>
That's a bug, not a feature. It means poor scaling as the number of
vcpus increases and as the number of devices increases.
Note nothing prevents steering multiple MSIs into a single vector. It's
a bad idea though.
> In addition, the signals and interrupts are priority aware, which is
> useful for things like 802.1p networking where you may establish 8-tx
> and 8-rx queues for your virtio-net device. x86 APIC really has no
> usable equivalent, so PCI is stuck here.
>
x86 APIC is priority aware.
> Also, the signals can be allocated on-demand for implementing things
> like IPC channels in response to guest requests since there is no
> assumption about device-to-interrupt mappings. This is more flexible.
>
Yes. However given that vectors are a scarce resource you're severely
limited in that. And if you're multiplexing everything on one vector,
then you can just as well demultiplex your channels in the virtio driver
code.
> And through all of this, this design would work in any guest even if it
> doesn't have PCI (e.g. lguest, UML, physical systems, etc).
>
That is true for virtio which works on pci-less lguest and s390.
> -- Bottom Line --
>
> The idea here is to generalize all the interesting parts that are common
> (fast sync+async io, context-switch mitigation, back-end models, memory
> abstractions, signal-path routing, etc) that a variety of linux based
> technologies can use (kvm, lguest, openvz, uml, physical systems) and
> only require the thin "connector" code to port the system around. The
> idea is to try to get this aspect of PV right once, and at some point in
> the future, perhaps vbus will be as ubiquitous as PCI. Well, perhaps
> not *that* ubiquitous, but you get the idea ;)
>
That is exactly the design goal of virtio (except it limits itself to
virtualization).
> Then device models like virtio can ride happily on top and we end up
> with a really robust and high-performance Linux-based stack. I don't
> buy the argument that we already have PCI so lets use it. I don't think
> its the best design and I am not afraid to make an investment in a
> change here because I think it will pay off in the long run.
>
Sorry, I don't think you've shown any quantifiable advantages.
--
error compiling committee.c: too many arguments to function
^ permalink raw reply
* Re: [PATCH v3 3/6] vbus: add a "vbus-proxy" bus model for vbus_driver objects
From: Avi Kivity @ 2009-08-18 7:38 UTC (permalink / raw)
To: Anthony Liguori
Cc: Gregory Haskins, Ingo Molnar, Gregory Haskins, kvm,
alacrityvm-devel, linux-kernel, netdev, Michael S. Tsirkin
In-Reply-To: <4A89FF08.30509@codemonkey.ws>
On 08/18/2009 04:08 AM, Anthony Liguori wrote:
> I believe strongly that we should avoid putting things in the kernel
> unless they absolutely have to be. I'm definitely interested in
> playing with vhost to see if there are ways to put even less in the
> kernel. In particular, I think it would be a big win to avoid
> knowledge of slots in the kernel by doing ring translation in
> userspace. This implies a userspace transition in the fast path.
> This may or may not be acceptable. I think this is going to be a very
> interesting experiment and will ultimately determine whether my
> intuition about the cost of dropping to userspace is right or wrong.
I believe with a perfectly scaling qemu this should be feasible.
Currently qemu is far from scaling perfectly, but inefficient userspace
is not a reason to put things into the kernel.
Having a translated ring is also a nice solution for migration -
userspace can mark the pages dirty while translating the receive ring.
Still, in-kernel translation is simple enough that I think we should
keep it.
--
error compiling committee.c: too many arguments to function
^ permalink raw reply
* Re: Receive side performance issue with multi-10-GigE and NUMA
From: Bill Fink @ 2009-08-18 7:07 UTC (permalink / raw)
To: Jesse Barnes
Cc: Brandeburg, Jesse, Neil Horman, Andrew Gallatin, Brice Goglin,
Linux Network Developers, Yinghai Lu
In-Reply-To: <20090817095302.0c41ef68@jbarnes-g45>
On Mon, 17 Aug 2009 09:53:02 -0700, Jesse Barnes wrote:
> On Fri, 14 Aug 2009 16:31:55 -0400
> Bill Fink <billfink@mindspring.com> wrote:
>
> Hm, yeah it probably should have the full CPU mask...
>
> > Also, is it just not possible on this type of Intel Xeon system to
> > properly associate the PCI devices with the nearest NUMA node?
>
> All the PCI devices hang off the root complex, which is the same
> distance to each node of memory (at least that's my understanding for
> current platforms).
I admit to being confused then. The basic system architecture
of the SuperMicro system is:
Memory----CPU1----QPI----CPU2----Memory
| |
| |
QPI QPI
| |
| |
5520----QPI----5520
|||| ||||
|||| ||||
|||| ||||
PCIe PCIe
It doesn't appear that a given PCIe device is equidistant to the
two nodes of memory. It's one QPI hop to the "local" (same side)
node, and two QPI hops to the "remote" (far side) node. But then
I don't know what a root complex is, and how it fits into the
system architecture above.
> > In any event, the patch didn't help (or hurt). The transmit
> > performance remained at ~100 Gbps while the receive performance
> > remained at 55 Gbps.
>
> Maybe the other Jesse has some ideas here.
Any and all ideas welcome. I even considered the idea that maybe
instead of transferring 9000 bytes of payload, perhaps it was
transferring the next higher power of 2, namely 16384, since
bc told me that 9000/16384*100 was 54.9316. But I tried a test
today with an MTU of 8000 and it didn't make any difference.
BTW here's a diff of an "lspci -vvvxxxx" on the better receive side
performing Asus system (<) versus on the SuperMicro system (>) for
one of the Myricom 10-GigE interfaces:
[root@xeontest1 ~]# diff -bw /tmp/foo2 /tmp/foo3
1c1
< 06:00.0 Ethernet controller: MYRICOM Inc. Myri-10G Dual-Protocol NIC (rev 01)
---
> 04:00.0 Ethernet controller: MYRICOM Inc. Myri-10G Dual-Protocol NIC (rev 01)
3c3
< Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
---
> Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
I don't know what the ParErr- versus ParErr+ means.
5,9c5,9
< Latency: 0, Cache Line Size: 64 bytes
< Interrupt: pin A routed to IRQ 2277
< Region 0: Memory at da000000 (64-bit, prefetchable) [size=16M]
< Region 2: Memory at fa900000 (64-bit, non-prefetchable) [size=1M]
< Expansion ROM at fa880000 [disabled] [size=512K]
---
> Latency: 0, Cache Line Size: 256 bytes
> Interrupt: pin A routed to IRQ 121
> Region 0: Memory at f3000000 (64-bit, prefetchable) [size=16M]
> Region 2: Memory at fa300000 (64-bit, non-prefetchable) [size=1M]
> Expansion ROM at fa280000 [disabled] [size=512K]
11c11
< Address: 00000000fee0400c Data: 4183
---
> Address: 00000000fee00000 Data: 40cc
45c45
< Capabilities: [1a8] Device Serial Number b6-be-46-ff-ff-dd-60-00
---
> Capabilities: [1a8] Device Serial Number 88-be-46-ff-ff-dd-60-00
I don't see much difference other than a larger Cache Line Size
on the SuperMicro system.
47,48c47,48
< 00: c1 14 08 00 06 05 10 00 01 00 00 02 10 00 00 00
< 10: 0c 00 00 da 00 00 00 00 04 00 90 fa 00 00 00 00
---
> 00: c1 14 08 00 46 05 10 00 01 00 00 02 40 00 00 00
> 10: 0c 00 00 f3 00 00 00 00 04 00 30 fa 00 00 00 00
50,52c50,52
< 30: 00 00 88 fa 44 00 00 00 00 00 00 00 0b 01 00 00
< 40: 00 00 00 00 05 54 81 00 0c 40 e0 fe 00 00 00 00
< 50: 83 41 00 00 01 5c 03 00 00 20 00 64 10 a0 02 00
---
> 30: 00 00 28 fa 44 00 00 00 00 00 00 00 0e 01 00 00
> 40: 00 00 00 00 05 54 81 00 00 00 e0 fe 00 00 00 00
> 50: cc 40 00 00 01 5c 03 00 00 20 00 64 10 a0 02 00
73c73
< 1a0: 00 00 00 00 00 00 00 00 03 00 01 00 b6 be 46 ff
---
> 1a0: 00 00 00 00 00 00 00 00 03 00 01 00 88 be 46 ff
And here's part of the dmesg output on the Asus system:
myri10ge: Version 1.4.3-1.358
myri10ge 0000:06:00.0: PCI INT A -> GSI 35 (level, low) -> IRQ 35
myri10ge 0000:06:00.0: setting latency timer to 64
mtrr: type mismatch for da000000,1000000 old: write-back new: write-combining
firmware: requesting myri10ge_eth_z8e.dat
myri10ge 0000:06:00.0: Not enabling ECRC on non-root port 0000:05:02.0
firmware: requesting myri10ge_eth_z8e.dat
myri10ge 0000:06:00.0: MSI IRQ 2282, tx bndry 4096, fw myri10ge_eth_z8e.dat, WC
Disabled
And on the SuperMicro system:
myri10ge: Version 1.4.4-1.401
alloc irq_desc for 35 on cpu 0 node 0
alloc kstat_irqs on cpu 0 node 0
myri10ge 0000:04:00.0: PCI INT A -> GSI 35 (level, low) -> IRQ 35
myri10ge 0000:04:00.0: setting latency timer to 64
myri10ge 0000:04:00.0: firmware: requesting myri10ge_eth_z8e.dat
myri10ge 0000:04:00.0: Not enabling ECRC on non-root port 0000:03:02.0
myri10ge 0000:04:00.0: firmware: requesting myri10ge_eth_z8e.dat
alloc irq_desc for 112 on cpu 0 node 0
alloc kstat_irqs on cpu 0 node 0
myri10ge 0000:04:00.0: irq 112 for MSI/MSI-X
myri10ge 0000:04:00.0: MSI IRQ 112, tx bndry 4096, fw myri10ge_eth_z8e.dat, WC E
nabled
alloc irq_desc for 24 on cpu 0 node 0
alloc kstat_irqs on cpu 0 node 0
Interestingly, the "WC Enabled" is only indicated on the first two
10-GigE interfaces and disabled on the other ten. For the Asus system
it indicates "WC Disabled" on all the interfaces, but also has that
earlier bit about "old: write-back new: write-combining", which doesn't
appear on the SuperMicro system (although that is using a slightly
newer version of the myri10ge driver).
-Thanks
-Bill
^ permalink raw reply
* Re: [PATCH] drivers/net: fixed drivers that support netpoll use ndo_start_xmit()
From: DDD @ 2009-08-18 5:59 UTC (permalink / raw)
To: David Miller, mpm; +Cc: netdev, linux-kernel, Bruce.Ashfield, jason.wessel
In-Reply-To: <20090814.163232.234331420.davem@davemloft.net>
On Fri, 2009-08-14 at 16:32 -0700, David Miller wrote:
> From: Matt Mackall <mpm@selenic.com>
> Date: Fri, 14 Aug 2009 10:33:39 -0500
>
> > On Fri, 2009-08-14 at 13:12 +0800, DDD wrote:
> >> The NETPOLL API requires that interrupts remain disabled in
> >> netpoll_send_skb(). The use of spin_lock_irq() and
> spin_unlock_irq()
> >> in the NETPOLL API callbacks causes the interrupts to get enabled
> and
> >> can lead to kernel instability.
> >>
> >> The solution is to use spin_lock_irqsave() and
> spin_unlock_restore()
> >> to prevent the irqs from getting enabled while in
> netpoll_send_skb().
> ...
> >> Signed-off-by: Dongdong Deng <dongdong.deng@windriver.com>
> >> Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
> >> Acked-by: Bruce Ashfield <bruce.ashfield@windriver.com>
> >
> > Acked-by: Matt Mackall <mpm@selenic.com>
>
> Applied, thanks everyone.
Thanks. :-)
>
> > Perhaps we should also have a WARN_ONCE if start_xmit returns with
> > interrupts enabled?
>
> Probably a good idea.
I think it is a good idea too, thanks for your suggestion, Matt.
I will do a tiny patch for it shortly.
Dongdong
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox