* Re: virtio: put last_used and last_avail index into ring itself.
From: Rusty Russell @ 2010-05-10 3:11 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Eric Dumazet, kvm, netdev, linux-kernel, virtualization, linux-mm,
s.hetze, hpa, Daniel Walker, mingo, akpm
In-Reply-To: <20100509085733.GD16775@redhat.com>
On Sun, 9 May 2010 06:27:33 pm Michael S. Tsirkin wrote:
> On Fri, May 07, 2010 at 12:35:39PM +0930, Rusty Russell wrote:
> > Then there's padding to page boundary. That puts us on a cacheline again
> > for the used ring; also 2 bytes per entry.
> >
>
> Hmm, is used ring really 2 bytes per entry?
Err, no, I am an idiot.
> /* u32 is used here for ids for padding reasons. */
> struct vring_used_elem {
> /* Index of start of used descriptor chain. */
> __u32 id;
> /* Total length of the descriptor chain which was used (written to) */
> __u32 len;
> };
>
> struct vring_used {
> __u16 flags;
> __u16 idx;
> struct vring_used_elem ring[];
> };
OK, now I get it. Sorry, I was focussed on the avail ring.
> I thought that used ring has 8 bytes per entry, and that struct
> vring_used is aligned at page boundary, this
> would mean that ring element is at offset 4 bytes from page boundary.
> Thus with cacheline size 128 bytes, each 4th element crosses
> a cacheline boundary. If we had a 4 byte padding after idx, each
> used element would always be completely within a single cacheline.
I think the numbers are: every 16th entry hits two cachelines. So currently
the first 15 entries are "free" (assuming we hit the idx cacheline anyway),
then 1 in 16 cost 2 cachelines. That makes the aligned version win when
N > 240.
But, we access the array linearly. So the extra cacheline cost is in fact
amortized. I doubt it could be measured, but maybe vring_get_buf() should
prefetch? While you're there, we could use an & rather than a mod on the
calculation, which may actually be measurable :)
Cheers,
Rusty.
^ permalink raw reply
* Re: [PATCH RFC] virtio: put last seen used index into ring itself
From: Michael S. Tsirkin @ 2010-05-09 21:06 UTC (permalink / raw)
To: Rusty Russell; +Cc: qemu-devel, linux-kernel, kvm, virtualization
In-Reply-To: <201005071303.29129.rusty@rustcorp.com.au>
On Fri, May 07, 2010 at 01:03:28PM +0930, Rusty Russell wrote:
> On Thu, 6 May 2010 03:49:46 pm Michael S. Tsirkin wrote:
> > Now, I also added an mb() in guest between read and write so
> > that last used index write can not get ahead of used index read.
> > It does feel good to have it there, but I can not say why
> > it's helpful. Works fine without it, but then these
> > subtle races might be hard to trigger. What do you think?
>
> I couldn't see that in the patch? I don't think it's necessary
> though, since the write of depends last_used depends on the read of
> used (and no platform we care about would reorder such a thing).
Well, there's no data dependency, is there?
> I'm reasonably happy, but we should write some convenient test for
> missing interrupts.
>
> I'm thinking of a sender which does a loop: blasts 1MB of UDP packets,
> then prints the time and sleep(1). The receiver would print the time
> every 1MB of received data. The two times should almost exactly correspond.
>
> Assuming that the network doesn't overflow and lose stuff, this should
> identify any missing wakeup/interrupts (depending on direction used).
>
> Cheers,
> Rusty.
^ permalink raw reply
* Re: virtio: put last_used and last_avail index into ring itself.
From: Michael S. Tsirkin @ 2010-05-09 8:57 UTC (permalink / raw)
To: Rusty Russell
Cc: Eric Dumazet, kvm, netdev, linux-kernel, virtualization, linux-mm,
s.hetze, hpa, Daniel Walker, mingo, akpm
In-Reply-To: <201005071235.40590.rusty@rustcorp.com.au>
On Fri, May 07, 2010 at 12:35:39PM +0930, Rusty Russell wrote:
> On Thu, 6 May 2010 03:57:55 pm Michael S. Tsirkin wrote:
> > On Thu, May 06, 2010 at 10:22:12AM +0930, Rusty Russell wrote:
> > > On Wed, 5 May 2010 03:52:36 am Michael S. Tsirkin wrote:
> > > > What do you think?
> > >
> > > I think everyone is settled on 128 byte cache lines for the forseeable
> > > future, so it's not really an issue.
> >
> > You mean with 64 bit descriptors we will be bouncing a cache line
> > between host and guest, anyway?
>
> I'm confused by this entire thread.
>
> Descriptors are 16 bytes. They are at the start, so presumably aligned to
> cache boundaries.
>
> Available ring follows that at 2 bytes per entry, so it's also packed nicely
> into cachelines.
>
> Then there's padding to page boundary. That puts us on a cacheline again
> for the used ring; also 2 bytes per entry.
>
Hmm, is used ring really 2 bytes per entry?
/* u32 is used here for ids for padding reasons. */
struct vring_used_elem {
/* Index of start of used descriptor chain. */
__u32 id;
/* Total length of the descriptor chain which was used (written to) */
__u32 len;
};
struct vring_used {
__u16 flags;
__u16 idx;
struct vring_used_elem ring[];
};
> I don't see how any change in layout could be more cache friendly?
> Rusty.
I thought that used ring has 8 bytes per entry, and that struct
vring_used is aligned at page boundary, this
would mean that ring element is at offset 4 bytes from page boundary.
Thus with cacheline size 128 bytes, each 4th element crosses
a cacheline boundary. If we had a 4 byte padding after idx, each
used element would always be completely within a single cacheline.
What am I missing?
--
MST
^ permalink raw reply
* Re: Patch to auto-load MSFT PV NIC driver
From: Jeremy Fitzhardinge @ 2010-05-08 23:02 UTC (permalink / raw)
To: Greg KH
Cc: devel@driverdev.osuosl.org, Virtualization@lists.osdl.org,
Ky Srinivasan, Haiyang Zhang
In-Reply-To: <20100508142716.GA32133@kroah.com>
On 05/08/2010 07:27 AM, Greg KH wrote:
> How would you do that? The symbol resolution will not allow hv_netvsc
> to be loaded without hv_vmbus being around.
>
> Now I do think that just adding the same pci/dmi ids to all of the
> modules is a bit odd, and that something else needs to be done here, but
> at the moment, I can't think of anything better.
>
Could you make hv_vmbus a real bus implementation, which knows how to
enumerate and probe devices hanging off it?
J
^ permalink raw reply
* Re: Patch to auto-load MSFT PV NIC driver
From: Greg KH @ 2010-05-08 14:27 UTC (permalink / raw)
To: Hank Janssen
Cc: devel@driverdev.osuosl.org, Virtualization@lists.osdl.org,
Ky Srinivasan, Haiyang Zhang
In-Reply-To: <8AFC7968D54FB448A30D8F38F259C5621AFFB3F9@TK5EX14MBXC114.redmond.corp.microsoft.com>
On Sat, May 08, 2010 at 01:52:01PM +0000, Hank Janssen wrote:
>
>
> >From: Ky Srinivasan [ksrinivasan@novell.com] - Friday, May 07, 2010 4:52 PM
> >
> > I am attaching a patch to auto-load hv_netvsc.
>
> I am not sure if this is the right approach. hv_netvsc takes a dependency on hv_vmbus.
> hv_vmbus does have the same DMI detection logic in it. But unless hv_vmbus has loaded
> up competely, hv_netvsc will fail on loadup. And I do not think we can guarantee that
> hv_vmbus has loaded yet.
Yes you can, the dependancies in the module will take care of it. Try
it, if you try to load the hv_netvsc module before hv_vmbus, modprobe
will load hv_vmbus first.
> It would be better to have netvsc check to make sure hv_vmbus is loaded.
How would you do that? The symbol resolution will not allow hv_netvsc
to be loaded without hv_vmbus being around.
Now I do think that just adding the same pci/dmi ids to all of the
modules is a bit odd, and that something else needs to be done here, but
at the moment, I can't think of anything better.
thanks,
greg k-h
^ permalink raw reply
* RE: Patch to auto-load MSFT PV NIC driver
From: Hank Janssen @ 2010-05-08 13:52 UTC (permalink / raw)
To: Ky Srinivasan, devel@driverdev.osuosl.org,
Virtualization@lists.osdl.org
Cc: Haiyang Zhang
In-Reply-To: <4BE453510200003000084906@sinclair.provo.novell.com>
>From: Ky Srinivasan [ksrinivasan@novell.com] - Friday, May 07, 2010 4:52 PM
>
> I am attaching a patch to auto-load hv_netvsc.
I am not sure if this is the right approach. hv_netvsc takes a dependency on hv_vmbus.
hv_vmbus does have the same DMI detection logic in it. But unless hv_vmbus has loaded
up competely, hv_netvsc will fail on loadup. And I do not think we can guarantee that
hv_vmbus has loaded yet.
It would be better to have netvsc check to make sure hv_vmbus is loaded.
Thanks,
Hank.
^ permalink raw reply
* Patch to auto-load MSFT PV NIC driver
From: Ky Srinivasan @ 2010-05-07 23:52 UTC (permalink / raw)
To: devel, Virtualization; +Cc: Haiyang Zhang
[-- Attachment #1: Type: text/plain, Size: 74 bytes --]
I am attaching a patch to auto-load hv_netvsc.
Regards,
K. Y
[-- Attachment #2: hv_netvsc_autoload.patch --]
[-- Type: text/plain, Size: 1905 bytes --]
From: K. Y. Srinivasan <ksrinivasan@novell.com>
Subject: Auto-load the hyperV PV net driver.
References: None
Patch-mainline:
Signed-off-by: K. Y. Srinivasan <ksrinivasan@novell.com>
Index: linux/drivers/staging/hv/netvsc_drv.c
===================================================================
--- linux.orig/drivers/staging/hv/netvsc_drv.c 2010-05-07 15:29:32.000000000 -0600
+++ linux/drivers/staging/hv/netvsc_drv.c 2010-05-07 15:30:00.000000000 -0600
@@ -29,6 +29,8 @@
#include <linux/etherdevice.h>
#include <linux/skbuff.h>
#include <linux/in.h>
+#include <linux/dmi.h>
+#include <linux/pci.h>
#include <net/arp.h>
#include <net/route.h>
#include <net/sock.h>
@@ -592,6 +594,20 @@ static int netvsc_drv_init(int (*drv_ini
return ret;
}
+static const struct dmi_system_id __initconst
+hv_netvsc_dmi_table[] __maybe_unused = {
+ {
+ .ident = "Hyper-V",
+ .matches = {
+ DMI_MATCH(DMI_SYS_VENDOR, "Microsoft Corporation"),
+ DMI_MATCH(DMI_PRODUCT_NAME, "Virtual Machine"),
+ DMI_MATCH(DMI_BOARD_NAME, "Virtual Machine"),
+ },
+ },
+ { },
+};
+MODULE_DEVICE_TABLE(dmi, hv_netvsc_dmi_table);
+
static int __init netvsc_init(void)
{
int ret;
@@ -599,6 +615,9 @@ static int __init netvsc_init(void)
DPRINT_ENTER(NETVSC_DRV);
DPRINT_INFO(NETVSC_DRV, "Netvsc initializing....");
+ if (!dmi_check_system(hv_netvsc_dmi_table))
+ return -ENODEV;
+
ret = netvsc_drv_init(NetVscInitialize);
DPRINT_EXIT(NETVSC_DRV);
@@ -613,6 +632,13 @@ static void __exit netvsc_exit(void)
DPRINT_EXIT(NETVSC_DRV);
}
+static const struct pci_device_id __initconst
+hv_netvsc_pci_table[] __maybe_unused = {
+ { PCI_DEVICE(0x1414, 0x5353) }, /* VGA compatible controller */
+ { 0 }
+};
+MODULE_DEVICE_TABLE(pci, hv_netvsc_pci_table);
+
module_param(netvsc_ringbuffer_size, int, S_IRUGO);
module_init(netvsc_init);
[-- Attachment #3: Type: text/plain, Size: 159 bytes --]
_______________________________________________
devel mailing list
devel@linuxdriverproject.org
http://driverdev.linuxdriverproject.org/mailman/listinfo/devel
^ permalink raw reply
* [Reminder] KVM Forum 2010: Call for Papers
From: KVM Forum 2010 Program Committee @ 2010-05-07 16:35 UTC (permalink / raw)
To: kvm, qemu-devel, virtualization, libvir-list
Just a reminder...The submission deadline is in one week.
thanks,
-KVM Forum 2010 Program Commitee
--
=================================================================
CALL FOR PAPERS
KVM Forum 2010
=================================================================
DESCRIPTION
The KVM Forum is back! After a break last year we're proud to present
this year's gathering around KVM again. The idea is to have everyone
involved with KVM development come together to talk about the future
and current state of KVM, teaching everyone some pieces of the puzzle
they might be missing without.
So if you're a KVM developer, mark the dates in your calendar! If
possible, also submit a talk -- we're interested in a wide variety of
KVM topics, so don't hesitate to propose a talk on your work.
If you're not a KVM developer, please read on nevertheless (or jump to
END USER COLLABORATION).
DATES / LOCATION
Conference: August 9 - 10, 2010
Location: Renaissance Boston Waterfront in Boston, MA
Abstracts due: May 14th, 2010
Notification: May 28th, 2010
Yes, we're colocated with LinuxCon. Tickets for the KVM Forum also count
for LinuxCon.
http://events.linuxfoundation.org/component/registrationpro/?func=details&did=34
PROCESS
At first check if it's before May 14th. If you're past that date, you're
out of luck. Now try to think hard and come up with a great idea that
you could talk about. Once you have that set, we need you to write up
a short abstract (~150 words) on it. In your submission please note
how long your talk will take. Slots vary in length up to one hour.
Also include in your proposal the proposal type -- one of: technical talk,
breakout session, or end-user talk. Add that information to the abstract
and submit it at the following URL:
http://events.linuxfoundation.org/cfp/cfp-add
Now, wait until May 24th. You will receive a notification on whether
your talk was accepted or not.
SCOPE OF TALKS
We have a list of suggested presentation topics below. These suggestions
are just for guidance, please feel free to submit a proposal on any
of these or related topics. In general, the more it's about backend
infrastructure, the better.
KVM
- Scaling and performance
- Nested virtualization
- I/O improvements
- Driver domains
- Time keeping
- Memory management (page sharing, swapping, huge pages, etc)
- Fault tolerance
- VEPA, vswitch
Embedded KVM
- KVM on ARM, PPC, MIPS, ...?
- Real-time requirements host/guest
- Device pass-through w/o iommu
- Custom device/platform models
QEMU
- Device model improvements
- New devices
- Security model
- Scaling and performance
- Desktop virtualization
- Increasing robustness
- Management interfaces
- QMP protocol and implementation
- Live migration
Virtio
- Speeding up existing devices
- Vhost
- Alternatives
- Using virtio in non-kvm environments
- Virtio on non-Linux
Management infrastructure
- Libvirt
- Kvm autotest
- Easy networking
- Qemud
BREAKOUT SESSION
We will reserve some time each day to break out for working sessions.
These sessions will be less formal than a presentation and more focused
on developing a solution to some real development issue. If you are
interested in getting developers together to hack on some code, submit
your proposal and just make it clear it's a breakout session proposal.
END-USER COLLABORATION
One of the big challenges as developers is to know what, where and how
people actually use our software. To solve this issue at least a little,
there will be a few slots reserved for end users talking about their
deployments, problems and achievements.
So if you have a KVM based deployment running in production or are about
to roll out one, please also submit a talk (see PROCESS), and simply
mark it asn an end-user collaboration proposal. We would love to have
an open discussion of fields where KVM/Qemu can still improve and you
would have the unique chance to steer that process!
Keep in mind that most of the Forum will be focused on development though,
so we suggest you also come with a good portion of technical interest :-).
And of course, no product marketing please! The purpose is to engage
with KVM developers.
LIGHTNING TALKS
In addition to submitted talks we will also have some room for lightning
talks. So if you have something you think might be done until the KVM
Forum, but you're not sure you could fill 15 minutes with it. Or if you
don't know if you'll make it until there, just keep in mind that you
will still get the chance to talk about it. Lightning talk submissions
and scheduling will be handled on-site at KVM Forum.
Thank you for your interest in KVM. We're looking forward to your
submissions and seeing you at the KVM Forum 2010 in August! Now, start
thinking about that talk you want to give.
Thanks,
your KVM Forum 2010 Program Commitee
Alexander Graf, Novell
Anthony Liguori, IBM
Avi Kivity, Red Hat
Chris Wright, Red Hat
Dor Laor, Red Hat
Jan Kiszka, Siemens
Please contact us with any questions or comments.
KVM-Forum-2010-PC@redhat.com
^ permalink raw reply
* [PATCH] virtio: initialize earlier
From: Rusty Russell @ 2010-05-07 4:58 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Stijn Tintel, virtualization
From: Stijn Tintel <stijn@linux-ipv6.be>
Move initialization of the virtio framework before the initialization of
mtd, so that block2mtd can be used on virtio-based block devices.
Addresses https://bugzilla.kernel.org/show_bug.cgi?id=15644
Signed-off-by: Stijn Tintel <stijn@linux-ipv6.be>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Cc: stable@kernel.org
---
drivers/Makefile | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/Makefile b/drivers/Makefile
index 34f1e10..f42a030 100644
--- a/drivers/Makefile
+++ b/drivers/Makefile
@@ -17,6 +17,7 @@ obj-$(CONFIG_SFI) += sfi/
obj-$(CONFIG_PNP) += pnp/
obj-$(CONFIG_ARM_AMBA) += amba/
+obj-$(CONFIG_VIRTIO) += virtio/
obj-$(CONFIG_XEN) += xen/
# regulators early, since some subsystems rely on them to initialize
@@ -108,7 +109,6 @@ obj-$(CONFIG_PPC_PS3) += ps3/
obj-$(CONFIG_OF) += of/
obj-$(CONFIG_SSB) += ssb/
obj-$(CONFIG_VHOST_NET) += vhost/
-obj-$(CONFIG_VIRTIO) += virtio/
obj-$(CONFIG_VLYNQ) += vlynq/
obj-$(CONFIG_STAGING) += staging/
obj-y += platform/
^ permalink raw reply related
* Re: [PATCH RFC] virtio: put last seen used index into ring itself
From: Rusty Russell @ 2010-05-07 3:33 UTC (permalink / raw)
To: Michael S. Tsirkin; +Cc: qemu-devel, linux-kernel, kvm, virtualization
In-Reply-To: <20100506061945.GA8363@redhat.com>
On Thu, 6 May 2010 03:49:46 pm Michael S. Tsirkin wrote:
> Now, I also added an mb() in guest between read and write so
> that last used index write can not get ahead of used index read.
> It does feel good to have it there, but I can not say why
> it's helpful. Works fine without it, but then these
> subtle races might be hard to trigger. What do you think?
I couldn't see that in the patch? I don't think it's necessary
though, since the write of depends last_used depends on the read of
used (and no platform we care about would reorder such a thing).
I'm reasonably happy, but we should write some convenient test for
missing interrupts.
I'm thinking of a sender which does a loop: blasts 1MB of UDP packets,
then prints the time and sleep(1). The receiver would print the time
every 1MB of received data. The two times should almost exactly correspond.
Assuming that the network doesn't overflow and lose stuff, this should
identify any missing wakeup/interrupts (depending on direction used).
Cheers,
Rusty.
^ permalink raw reply
* Re: [Qemu-devel] [PATCH RFC] virtio: put last seen used index into ring itself
From: Rusty Russell @ 2010-05-07 3:23 UTC (permalink / raw)
To: Avi Kivity
Cc: qemu-devel, virtualization, linux-kernel, kvm, Michael S. Tsirkin
In-Reply-To: <4BE29320.5090506@redhat.com>
On Thu, 6 May 2010 07:30:00 pm Avi Kivity wrote:
> On 05/05/2010 11:58 PM, Michael S. Tsirkin wrote:
> > + /* We publish the last-seen used index at the end of the available ring.
> > + * It is at the end for backwards compatibility. */
> > + vr->last_used_idx =&(vr)->avail->ring[num];
> > + /* Verify that last used index does not spill over the used ring. */
> > + BUG_ON((void *)vr->last_used_idx +
> > + sizeof *vr->last_used_idx> (void *)vr->used);
> > }
> >
>
> Shouldn't this be on its own cache line?
It's next to the available ring; because that's where the guest publishes
its data. That whole page is guest-write, host-read.
Putting it on a cacheline by itself would be a slight pessimization; the host
cpu would have to get the last_used_idx cacheline and the avail descriptor
cacheline every time. This way, they are sometimes the same cacheline.
Hope that clarifies,
Rusty.
^ permalink raw reply
* Re: virtio: put last_used and last_avail index into ring itself.
From: Rusty Russell @ 2010-05-07 3:05 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Eric Dumazet, kvm, netdev, linux-kernel, virtualization, linux-mm,
s.hetze, hpa, Daniel Walker, mingo, akpm
In-Reply-To: <20100506062755.GC8363@redhat.com>
On Thu, 6 May 2010 03:57:55 pm Michael S. Tsirkin wrote:
> On Thu, May 06, 2010 at 10:22:12AM +0930, Rusty Russell wrote:
> > On Wed, 5 May 2010 03:52:36 am Michael S. Tsirkin wrote:
> > > What do you think?
> >
> > I think everyone is settled on 128 byte cache lines for the forseeable
> > future, so it's not really an issue.
>
> You mean with 64 bit descriptors we will be bouncing a cache line
> between host and guest, anyway?
I'm confused by this entire thread.
Descriptors are 16 bytes. They are at the start, so presumably aligned to
cache boundaries.
Available ring follows that at 2 bytes per entry, so it's also packed nicely
into cachelines.
Then there's padding to page boundary. That puts us on a cacheline again
for the used ring; also 2 bytes per entry.
I don't see how any change in layout could be more cache friendly?
Rusty.
^ permalink raw reply
* Re: [Pv-drivers] RFC: Network Plugin Architecture (NPA) for vmxnet3
From: Christoph Hellwig @ 2010-05-06 20:21 UTC (permalink / raw)
To: Stephen Hemminger
Cc: Pankaj Thakkar, Dmitry Torokhov, pv-drivers@vmware.com,
netdev@vger.kernel.org, linux-kernel@vger.kernel.org,
virtualization@lists.linux-foundation.org, Christoph Hellwig
In-Reply-To: <20100505105253.0a8bc465@nehalam>
On Wed, May 05, 2010 at 10:52:53AM -0700, Stephen Hemminger wrote:
> Let me put it bluntly. Any design that allows external code to run
> in the kernel is not going to be accepted. Out of tree kernel modules are enough
> of a pain already, why do you expect the developers to add another
> interface.
Exactly. Until our friends at VMware get this basic fact it's useless
to continue arguing.
Pankaj and Dmitry: you're fine to waste your time on this, but it's not
going to go anywhere until you address that fundamental problem. The
first thing you need to fix in your archicture is to integrate the VF
function code into the kernel tree, and we can work from there.
Please post patches doing this if you want to resume the discussion.
^ permalink raw reply
* Re: [Pv-drivers] RFC: Network Plugin Architecture (NPA) for vmxnet3
From: Christoph Hellwig @ 2010-05-06 20:19 UTC (permalink / raw)
To: Pankaj Thakkar
Cc: Dmitry Torokhov, pv-drivers@vmware.com, netdev@vger.kernel.org,
linux-kernel@vger.kernel.org,
virtualization@lists.linux-foundation.org, Christoph Hellwig
In-Reply-To: <20100506180411.GC25364@vmware.com>
On Thu, May 06, 2010 at 11:04:11AM -0700, Pankaj Thakkar wrote:
> Plugin is x86 or x64 machine code. You write the plugin in C and compile it using gcc/ld to get the object file, we map the relevant sections only to the OS space.
Which is simply not supportable for a cross-platform operating system
like Linux.
^ permalink raw reply
* Re: [Pv-drivers] RFC: Network Plugin Architecture (NPA) for vmxnet3
From: Christoph Hellwig @ 2010-05-06 20:17 UTC (permalink / raw)
To: Pankaj Thakkar
Cc: Dmitry Torokhov, pv-drivers@vmware.com, netdev@vger.kernel.org,
linux-kernel@vger.kernel.org,
virtualization@lists.linux-foundation.org, Christoph Hellwig
In-Reply-To: <F1354E79A137A24CBA60059AA65CB1B802A235C535@EXCH-MBX-2.vmware.com>
On Wed, May 05, 2010 at 10:47:10AM -0700, Pankaj Thakkar wrote:
> > Forget about the licensing. Loading binary blobs written to a shim
> > layer is a complete pain in the ass and totally unsupportable, and
> > also uninteresting because of the overhead.
>
> [PT] Why do you think it is unsupportable? How different is it from any module written against a well maintained interface? What overhead are you talking about?
We only support in-kernel drivers, everything else is subject to changes
in the kernel API and ABI. What you do is basically introducing another
wrapper layer not allowing full access to the normal Linux API. People
have tried this before and we're not willing to add it. Do a little
research on Project UDI if you're curious.
> > (1) move the limited VF drivers directly into the kernel tree,
> > talk to them through a normal ops vector
> [PT] This assumes that all the VF drivers would always be available.
Yes, absolutely. Just as we assume that for every other driver.
> Also we have to support windows and our current design supports it nicely in an OS agnostic manner.
And that's not something we care about at all. The Linux kernel has
traditionally a very hostile position against cross platform drivers for
reasons well explained before at many occasions.
> > (2) get rid of the whole shim crap and instead integrate the limited
> > VF driver with the full VF driver we already have, instead of
> > duplicating the code
> [PT] Having a full VF driver adds a lot of dependency on the guest VM and this is what NPA tries to avoid.
Yes, of course it does. It's a normal driver at the point which it
should have been from day one.
> > (3) don't make the PV to VF integration VMware-specific but also
> > provide an open reference implementation like virtio. We're not
> > going to add massive amount of infrastructure that is not actually
> > useable in a free software stack.
> [PT] Today this is tied to vmxnet3 device and is intended to work on ESX hypervisor only (vmxnet3 works on VMware hypervisor only). All the loading support is inside the ESX hypervisor. I am going to post the interface between the shell and the plugin soon and you can see that there is not a whole lot of dependency or infrastructure requirements from the Linux kernel. Please keep in mind that we don't use Linux as a hypervisor but as a guest VM.
But we use Linux as the hypervisor, too. So if you want to target a
major infrastructure you might better make it available for that case.
^ permalink raw reply
* Re: [Pv-drivers] RFC: Network Plugin Architecture (NPA) for vmxnet3
From: Pankaj Thakkar @ 2010-05-06 18:04 UTC (permalink / raw)
To: Gleb Natapov
Cc: Dmitry Torokhov, pv-drivers@vmware.com, netdev@vger.kernel.org,
linux-kernel@vger.kernel.org,
virtualization@lists.linux-foundation.org, Christoph Hellwig
In-Reply-To: <20100506081933.GF10044@redhat.com>
On Thu, May 06, 2010 at 01:19:33AM -0700, Gleb Natapov wrote:
> Overhead of interpreting bytecode plugin is written in. Or are you
> saying plugin is x86 assembly (32bit or 64bit btw?) and other arches
> will have to have in kernel x86 emulator to use the plugin (like some
> of them had for vgabios)?
>
Plugin is x86 or x64 machine code. You write the plugin in C and compile it using gcc/ld to get the object file, we map the relevant sections only to the OS space.
NPA is a way of enabling passthrough of SR-IOV NICs with live migration support on ESX Hypervisor which runs only on x86/x64 hardware. It only supports x86/x64 guest OS. So we don't have to worry about other architectures. If NPA approach needs to be extended and adopted by other hypervisors then we have to take care of that. Today we have two plugins images per VF (one for 32-bit, one for 64-bit).
^ permalink raw reply
* Re: [Qemu-devel] question on virtio
From: Jamie Lokier @ 2010-05-06 16:43 UTC (permalink / raw)
To: Michael S. Tsirkin; +Cc: qemu-devel, kvm, virtualization
In-Reply-To: <20100505110947.GA27872@redhat.com>
Michael S. Tsirkin wrote:
> Hi!
> I see this in virtio_ring.c:
>
> /* Put entry in available array (but don't update avail->idx *
> until they do sync). */
>
> Why is it done this way?
> It seems that updating the index straight away would be simpler, while
> this might allow the host to specilatively look up the buffer and handle
> it, without waiting for the kick.
Even better, if the host updates a location containing which index it
has seen recently, you can avoid the kick entirely during sustained
flows - just like your recent patch to avoid sending irqs to the
guest.
-- Jamie
^ permalink raw reply
* Re: [Qemu-devel] Re: [PATCH] virtio-spec: document block CMD and FLUSH
From: Jamie Lokier @ 2010-05-06 15:25 UTC (permalink / raw)
To: Rusty Russell
Cc: tytso, kvm, Michael S. Tsirkin, Neil Brown, qemu-devel,
virtualization, Jens Axboe, hch
In-Reply-To: <201005051428.41735.rusty@rustcorp.com.au>
Rusty Russell wrote:
> On Wed, 5 May 2010 05:47:05 am Jamie Lokier wrote:
> > Jens Axboe wrote:
> > > On Tue, May 04 2010, Rusty Russell wrote:
> > > > ISTR someone mentioning a desire for such an API years ago, so CC'ing the
> > > > usual I/O suspects...
> > >
> > > It would be nice to have a more fuller API for this, but the reality is
> > > that only the flush approach is really workable. Even just strict
> > > ordering of requests could only be supported on SCSI, and even there the
> > > kernel still lacks proper guarantees on error handling to prevent
> > > reordering there.
> >
> > There's a few I/O scheduling differences that might be useful:
> >
> > 1. The I/O scheduler could freely move WRITEs before a FLUSH but not
> > before a BARRIER. That might be useful for time-critical WRITEs,
> > and those issued by high I/O priority.
>
> This is only because noone actually wants flushes or barriers, though
> I/O people seem to only offer that. We really want "<these writes> must
> occur before <this write>". That offers maximum choice to the I/O subsystem
> and potentially to smart (virtual?) disks.
We do want flushes for the "D" in ACID - such things as after
receiving a mail, or blog update into a database file (could be TDB),
and confirming that to the sender, to have high confidence that the
update won't disappear on system crash or power failure.
Less obviously, it's also needed for the "C" in ACID when more than
one file is involved. "C" is about differently updated things staying
consistent with each other.
For example, imagine you have a TDB file mapping Samba usernames to
passwords, and another mapping Samba usernames to local usernames. (I
don't know if you do this; it's just an illustration).
To rename a Samba user involves updating both. Let's ignore transient
transactional issues :-) and just think about what happens with
per-file barriers and no sync, when a crash happens long after the
updates, and before the system has written out all data and issued low
level cache flushes.
After restarting, due to lack of sync, the Samba username could be
present in one file and not the other.
> > 2. The I/O scheduler could move WRITEs after a FLUSH if the FLUSH is
> > only for data belonging to a particular file (e.g. fdatasync with
> > no file size change, even on btrfs if O_DIRECT was used for the
> > writes being committed). That would entail tagging FLUSHes and
> > WRITEs with a fs-specific identifier (such as inode number), opaque
> > to the scheduler which only checks equality.
>
> This is closer. In userspace I'd be happy with a "all prior writes to this
> struct file before all future writes". Even if the original guarantees were
> stronger (ie. inode basis). We currently implement transactions using 4 fsync
> /msync pairs.
>
> write_recovery_data(fd);
> fsync(fd);
> msync(mmap);
> write_recovery_header(fd);
> fsync(fd);
> msync(mmap);
> overwrite_with_new_data(fd);
> fsync(fd);
> msync(mmap);
> remove_recovery_header(fd);
> fsync(fd);
> msync(mmap);
>
> Yet we really only need ordering, not guarantees about it actually hitting
> disk before returning.
>
> > In other words, FLUSH can be more relaxed than BARRIER inside the
> > kernel. It's ironic that we think of fsync as stronger than
> > fbarrier outside the kernel :-)
>
> It's an implementation detail; barrier has less flexibility because it has
> less information about what is required. I'm saying I want to give you as
> much information as I can, even if you don't use it yet.
I agree, and I've started a few threads about it over the last couple of years.
An fsync_range() system call would be very easy to use and
(most importantly) easy to understand.
With optional flags to weaken it (into fdatasync, barrier without sync,
sync without barrier, one-sided barrier, no lowlevel cache-flush, don't rush,
etc.), it would be very versatile, and still easy to understand.
With an AIO version, and another flag meaning don't rush, just return
when satisfied, and I suspect it would be useful for the most
demanding I/O apps.
-- Jamie
^ permalink raw reply
* Re: [Qemu-devel] Re: [PATCH] virtio-spec: document block CMD and FLUSH
From: Jamie Lokier @ 2010-05-06 14:57 UTC (permalink / raw)
To: Rusty Russell
Cc: tytso, kvm, Michael S. Tsirkin, Neil Brown, qemu-devel,
virtualization, Jens Axboe, hch
In-Reply-To: <201005061535.20619.rusty@rustcorp.com.au>
Rusty Russell wrote:
> > Seems over-zealous.
> > If the recovery_header held a strong checksum of the recovery_data you would
> > not need the first fsync, and as long as you have two places to write recovery
> > data, you don't need the 3rd and 4th syncs.
> > Just:
> > write_internally_checksummed_recovery_data_and_header_to_unused_log_space()
> > fsync / msync
> > overwrite_with_new_data()
> >
> > To recovery you choose the most recent log_space and replay the content.
> > That may be a redundant operation, but that is no loss.
>
> I think you missed a checksum for the new data? Otherwise we can't tell if
> the new data is completely written.
The data checksum can go in the recovery-data block. If there's
enough slack in the log, by the time that recovery-data block is
overwritten, you can be sure that an fsync has been done for that
data (by a later commit).
> But yes, I will steal this scheme for TDB2, thanks!
Take a look at the filesystems. I think ext4 did some optimisations
in this area, and that checksums had to be added anyway due to a
subtle replay-corruption problem that happens when the log is
partially corrupted, and followed by non-corrupt blocks.
Also, you can remove even more fsyncs by adding a bit of slack to the
data space and writing into unused/fresh areas some of the time -
i.e. a bit like btrfs/zfs or anything log-structured, but you don't
have to go all the way with that.
> In practice, it's the first sync which is glacial, the rest are pretty cheap.
The 3rd and 4th fsyncs imply a disk seek each, just because the
preceding writes are to different areas of the disk. Seeks are quite
slow - but not as slow as ext3 fsyncs :-) What do you mean by cheap?
That it's only a couple of seeks, or that you don't see even that?
>
> > Also cannot see the point of msync if you have already performed an fsync,
> > and if there is a point, I would expect you to call msync before
> > fsync... Maybe there is some subtlety there that I am not aware of.
>
> I assume it's this from the msync man page:
>
> msync() flushes changes made to the in-core copy of a file that was
> mapped into memory using mmap(2) back to disk. Without use of this
> call there is no guarantee that changes are written back before mun‐
> map(2) is called.
Historically, that means msync() ensures dirty mapping data is written
to the file as if with write(), and that mapping pages are removed or
refreshed to get the effect of read() (possibly a lazy one). It's
more obvious in the early mmap implementations where mappings don't
share pages with the filesystem cache, so msync() has explicit
behaviour.
Like with write(), after calling msync() you would then call fsync()
to ensure the data is flushed to disk.
If you've been calling fsync then msync, I guess that's another fine
example of how these function are so hard to test, that they aren't.
Historically on Linux, msync has been iffy on some architectures, and
I'm still not sure it has the same semantics as other unixes. fsync
as we know has also been iffy, and even now that fsync is tidier it
does not always issue a hardware-level cache commit.
But then historically writable mmap has been iffy on a boatload of
unixes.
> > > It's an implementation detail; barrier has less flexibility because it has
> > > less information about what is required. I'm saying I want to give you as
> > > much information as I can, even if you don't use it yet.
> >
> > Only we know that approach doesn't work.
> > People will learn that they don't need to give the extra information to still
> > achieve the same result - just like they did with ext3 and fsync.
> > Then when we improve the implementation to only provide the guarantees that
> > you asked for, people will complain that they are getting empty files that
> > they didn't expect.
>
> I think that's an oversimplification: IIUC that occurred to people *not*
> using fsync(). They weren't using it because it was too slow. Providing
> a primitive which is as fast or faster and more specific doesn't have the
> same magnitude of social issues.
I agree with Rusty. Let's make it perform well so there is no reason
to deliberately avoid using it, and let's make say what apps actually
want to request without being way too strong.
And please, if anyone has ideas on how we could make correct use of
these functions *testable* by app authors, I'm all ears. Right now it
is quite difficult - pulling power on hard disks mid-transaction is
not a convenient method :)
> > The abstraction I would like to see is a simple 'barrier' that contains no
> > data and has a filesystem-wide effect.
>
> I think you lack ambition ;)
>
> Thinking about the single-file use case (eg. kvm guest or tdb), isn't that
> suboptimal for md? Since you have to hand your barrier to every device
> whereas a file-wide primitive may theoretically only go to some.
Yes.
Note that database-like programs still need fsync-like behaviour
*sometimes*: The "D" in ACID depends on it, and the "C" in ACID also
depends on it where multiple files are involved which must contain
consistent data with each other after crash/recovery (Perhaps Samba
depends on this?)
Single-file sync is valuable just like single-file barrier, and so is
the combination.
Since you mentioned ambition, think about multi-file updates. They're
analogous in userspace to MD's barrier/sync requirements in
kernelspace.
One API that supports multi-file update barriers is "long aio-fsync":
Something which returns when the data in earlier writes (to one file)
is committed, but does not force the commit to happen more quickly
than normal. Both single-file barriers (like you want for TDB) and
multi-file barriers can be implemented on top of that, but it's much
more difficult to use than an fbarrier() syscall, which is only
suitable for single-file. But I wonder if there would be many users
of fbarrier() who aren't perfectly capable of using something else if
needed.
-- Jamie
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linux-foundation.org/mailman/listinfo/virtualization
^ permalink raw reply
* Re: [Qemu-devel] [PATCH RFC] virtio: put last seen used index into ring itself
From: Avi Kivity @ 2010-05-06 10:00 UTC (permalink / raw)
To: Michael S. Tsirkin; +Cc: qemu-devel, linux-kernel, kvm, virtualization
In-Reply-To: <20100505205814.GA7090@redhat.com>
On 05/05/2010 11:58 PM, Michael S. Tsirkin wrote:
> + /* We publish the last-seen used index at the end of the available ring.
> + * It is at the end for backwards compatibility. */
> + vr->last_used_idx =&(vr)->avail->ring[num];
> + /* Verify that last used index does not spill over the used ring. */
> + BUG_ON((void *)vr->last_used_idx +
> + sizeof *vr->last_used_idx> (void *)vr->used);
> }
>
Shouldn't this be on its own cache line?
One way to achieve this is to allocate it at the end.
--
error compiling committee.c: too many arguments to function
^ permalink raw reply
* Re: RFC: Network Plugin Architecture (NPA) for vmxnet3
From: Avi Kivity @ 2010-05-06 8:58 UTC (permalink / raw)
To: Pankaj Thakkar
Cc: pv-drivers@vmware.com, netdev@vger.kernel.org,
linux-kernel@vger.kernel.org,
virtualization@lists.linux-foundation.org
In-Reply-To: <20100505194417.GV8323@vmware.com>
On 05/05/2010 10:44 PM, Pankaj Thakkar wrote:
> On Wed, May 05, 2010 at 10:59:51AM -0700, Avi Kivity wrote:
>
>> Date: Wed, 5 May 2010 10:59:51 -0700
>> From: Avi Kivity<avi@redhat.com>
>> To: Pankaj Thakkar<pthakkar@vmware.com>
>> CC: "linux-kernel@vger.kernel.org"<linux-kernel@vger.kernel.org>,
>> "netdev@vger.kernel.org"<netdev@vger.kernel.org>,
>> "virtualization@lists.linux-foundation.org"
>> <virtualization@lists.linux-foundation.org>,
>> "pv-drivers@vmware.com"<pv-drivers@vmware.com>,
>> Shreyas Bhatewara<sbhatewara@vmware.com>
>> Subject: Re: RFC: Network Plugin Architecture (NPA) for vmxnet3
>>
>> On 05/05/2010 02:02 AM, Pankaj Thakkar wrote:
>>
>>> 2. Hypervisor control: All control operations from the guest such as programming
>>> MAC address go through the hypervisor layer and hence can be subjected to
>>> hypervisor policies. The PF driver can be further used to put policy decisions
>>> like which VLAN the guest should be on.
>>>
>>>
>> Is this enforced? Since you pass the hardware through, you can't rely
>> on the guest actually doing this, yes?
>>
> We don't pass the whole VF to the guest. Only the BAR which is responsible for
> TX/RX/intr is mapped into guest space.
Does the SR/IOV spec guarantee that you will have such a separation?
>
>>
>>> We have reworked our existing Linux vmxnet3 driver to accomodate NPA by
>>> splitting the driver into two parts: Shell and Plugin. The new split driver is
>>>
>>>
>> So the Shell would be the reworked or new bond driver, and Plugins would
>> be ordinary Linux network drivers.
>>
> In NPA we do not rely on the guest OS to provide any of these services like
> bonding or PCI hotplug.
Well the Shell does some sort of bonding (there are two links and the
shell selects which one to exercise) and some sort of hotplug. Since
the Shell is part of the guest OS, you do rely on it.
It's certainly simpler than PCI hotplug or ordinary bonding.
> We don't rely on the guest OS to unmap a VF and switch
> a VM out of passthrough. In a bonding approach that becomes an issue you can't
> just yank a device from underneath, you have to wait for the OS to process the
> request and switch from using VF to the emulated device and this makes the
> hypervisor dependent on the guest OS.
How can you unmap the VF without guest cooperation? If you're executing
Plugin code, you can't yank anything out.
Are plugins executed with preemption/interrupts disabled?
> Also we don't rely on the presence of all
> the drivers inside the guest OS (be it Linux or Windows), the ESX hypervisor
> carries all the plugins and the PF drivers and injects the right one as needed.
> These plugins are guest agnostic and the IHVs do not have to write plugins for
> different OS.
>
What ISAs do those plugins support?
--
error compiling committee.c: too many arguments to function
^ permalink raw reply
* Re: [Pv-drivers] RFC: Network Plugin Architecture (NPA) for vmxnet3
From: Gleb Natapov @ 2010-05-06 8:19 UTC (permalink / raw)
To: Pankaj Thakkar
Cc: Dmitry Torokhov, pv-drivers@vmware.com, netdev@vger.kernel.org,
linux-kernel@vger.kernel.org,
virtualization@lists.linux-foundation.org, Christoph Hellwig
In-Reply-To: <F1354E79A137A24CBA60059AA65CB1B802A235C535@EXCH-MBX-2.vmware.com>
On Wed, May 05, 2010 at 10:47:10AM -0700, Pankaj Thakkar wrote:
>
>
> > -----Original Message-----
> > From: Christoph Hellwig [mailto:hch@infradead.org]
> > Sent: Wednesday, May 05, 2010 10:40 AM
> > To: Dmitry Torokhov
> > Cc: Christoph Hellwig; pv-drivers@vmware.com; Pankaj Thakkar;
> > netdev@vger.kernel.org; linux-kernel@vger.kernel.org;
> > virtualization@lists.linux-foundation.org
> > Subject: Re: [Pv-drivers] RFC: Network Plugin Architecture (NPA) for
> > vmxnet3
> >
> > On Wed, May 05, 2010 at 10:35:28AM -0700, Dmitry Torokhov wrote:
> > > Yes, with the exception that the only body of code that will be
> > > accepted by the shell should be GPL-licensed and thus open and
> > available
> > > for examining. This is not different from having a standard kernel
> > > module that is loaded normally and plugs into a certain subsystem.
> > > The difference is that the binary resides not on guest filesystem
> > > but elsewhere.
> >
> > Forget about the licensing. Loading binary blobs written to a shim
> > layer is a complete pain in the ass and totally unsupportable, and
> > also uninteresting because of the overhead.
>
> [PT] Why do you think it is unsupportable? How different is it from any module
> written against a well maintained interface? What overhead are you talking about?
>
Overhead of interpreting bytecode plugin is written in. Or are you
saying plugin is x86 assembly (32bit or 64bit btw?) and other arches
will have to have in kernel x86 emulator to use the plugin (like some
of them had for vgabios)?
--
Gleb.
^ permalink raw reply
* Re: [GIT PULL] amended: first round of vhost-net enhancements for net-next
From: David Miller @ 2010-05-06 7:29 UTC (permalink / raw)
To: mst; +Cc: kvm, virtualization, netdev, linux-kernel
In-Reply-To: <20100504112100.GA22172@redhat.com>
From: "Michael S. Tsirkin" <mst@redhat.com>
Date: Tue, 4 May 2010 14:21:01 +0300
> This is an amended pull request: I have rebased the tree to the
> correct patches. This has been through basic tests and seems
> to work fine here.
>
> The following tree includes a couple of enhancements that help vhost-net.
> Please pull them for net-next. Another set of patches is under
> debugging/testing and I hope to get them ready in time for 2.6.35,
> so there may be another pull request later.
Pulled, thanks Michael.
^ permalink raw reply
* RE: [Pv-drivers] RFC: Network Plugin Architecture (NPA) for vmxnet3
From: Shreyas Bhatewara @ 2010-05-06 7:25 UTC (permalink / raw)
To: Scott Feldman, Arnd Bergmann, Dmitry Torokhov
Cc: Pankaj Thakkar, pv-drivers@vmware.com, netdev@vger.kernel.org,
linux-kernel@vger.kernel.org,
virtualization@lists.linux-foundation.org, Christoph Hellwig
In-Reply-To: <C807719A.2E910%scofeldm@cisco.com>
> -----Original Message-----
> From: Scott Feldman [mailto:scofeldm@cisco.com]
> Sent: Wednesday, May 05, 2010 7:04 PM
> To: Shreyas Bhatewara; Arnd Bergmann; Dmitry Torokhov
> Cc: Christoph Hellwig; pv-drivers@vmware.com; netdev@vger.kernel.org;
> linux-kernel@vger.kernel.org; virtualization@lists.linux-
> foundation.org; Pankaj Thakkar
> Subject: Re: [Pv-drivers] RFC: Network Plugin Architecture (NPA) for
> vmxnet3
>
> On 5/5/10 10:29 AM, "Dmitry Torokhov" <dtor@vmware.com> wrote:
>
> > It would not be a binary blob but software properly released under
> GPL.
> > The current plan is for the shell to enforce GPL requirement on the
> > plugin code, similar to what module loaded does for regular kernel
> > modules.
>
> On 5/5/10 3:05 PM, "Shreyas Bhatewara" <sbhatewara@vmware.com> wrote:
>
> > The plugin image is not linked against Linux kernel. It is OS
> agnostic infact
> > (Eg. same plugin works for Linux and Windows VMs)
>
> Are there any issues with injecting the GPL-licensed plug-in into the
> Windows vmxnet3 NDIS driver?
>
> -scott
^ permalink raw reply
* RE: [Pv-drivers] RFC: Network Plugin Architecture (NPA) for vmxnet3
From: Shreyas Bhatewara @ 2010-05-06 7:25 UTC (permalink / raw)
To: Scott Feldman, Arnd Bergmann, Dmitry Torokhov
Cc: Pankaj Thakkar, pv-drivers@vmware.com, netdev@vger.kernel.org,
linux-kernel@vger.kernel.org,
virtualization@lists.linux-foundation.org, Christoph Hellwig
In-Reply-To: <C807719A.2E910%scofeldm@cisco.com>
> -----Original Message-----
> From: Scott Feldman [mailto:scofeldm@cisco.com]
> Sent: Wednesday, May 05, 2010 7:04 PM
> To: Shreyas Bhatewara; Arnd Bergmann; Dmitry Torokhov
> Cc: Christoph Hellwig; pv-drivers@vmware.com; netdev@vger.kernel.org;
> linux-kernel@vger.kernel.org; virtualization@lists.linux-
> foundation.org; Pankaj Thakkar
> Subject: Re: [Pv-drivers] RFC: Network Plugin Architecture (NPA) for
> vmxnet3
>
> On 5/5/10 10:29 AM, "Dmitry Torokhov" <dtor@vmware.com> wrote:
>
> > It would not be a binary blob but software properly released under
> GPL.
> > The current plan is for the shell to enforce GPL requirement on the
> > plugin code, similar to what module loaded does for regular kernel
> > modules.
>
> On 5/5/10 3:05 PM, "Shreyas Bhatewara" <sbhatewara@vmware.com> wrote:
>
> > The plugin image is not linked against Linux kernel. It is OS
> agnostic infact
> > (Eg. same plugin works for Linux and Windows VMs)
>
> Are there any issues with injecting the GPL-licensed plug-in into the
> Windows vmxnet3 NDIS driver?
>
> -scott
Scott,
Thanks for pointing out. This issue can be resolved by adding exception to the plugin license which allows it to link to a non-free program .(http://www.gnu.org/licenses/gpl-faq.html#GPLPluginsInNF)
->Shreyas
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox