[Qemu-devel] virtio-9p

[Qemu-devel] virtio-9p

[Linda]

Hello,
     I am a summer intern under the Outreachy program (returning to 
software development after a long hiatus).  I am developing a prototype 
of a xen front/back end for a 9p file transport.
     Since virtio is the most complete, and used, transport, I am using 
that as a model.
     I am encountering some problems that, to fix sanely, will require 
making changes to virtio code.  On the plus side, when I'm finished with 
this project, any transport will be able to use the 9P server, with no 
modification.
     As background, for the backend, I have been looking at the code, 
written by Anthony Liguori, and maintained by Aneesh Kumar (who I sent 
this email to, earlier).  It looks to me (please correct me if I'm 
wrong, on this or any other point, below) as if Anthony wrote not just a 
backend transport layer, but the server as well.  AFAICT, there is no 
other Linux 9p server.
     virtio-9p.c contains a lot of this server code, the rest spread 
between 13 other files which handle all file access operations, 
converting them from 9p to Linux file system calls.
     virtio-9p.c also contains some virtio-specific code (although most 
of that is in virtio-device.c).

The problems I am encountering are the following:

1.  In the virio-9p.h is a struct V9fsPDU that contains an element (in 
the middle of the struct) of type VirtQueueElement. Every 9p I/O command 
handler, as well as co-routines and support functions that go with them 
(i.e., a large part of the server), passes a parameter that is a struct 
V9fsPDU.   Almost all of these use only the variable that defines state 
information, and never touch the VirtQueueElement member.
     The easiest fix for this is to have a separate header file with a 
#define GENERIC_9P_SERVER
     Then I could modify the virtio-9p.h with:
             #ifdef GENERIC_9P_SERVER
                    a union of a void *, a char * (what I use), and a 
VirtQueueElement (guaranteeing the size is unchanged)
             #else
                     VirtQueueElement    elem;
             #endif

     It's not my favorite construct, but it would involve the least 
amount of changes to the code.   Before I modify a header file, that 
code, I'm not touching, is dependent on, I wanted to know if this is an 
OK way.  If not, is there another way (short of copying fourteen files, 
and changing the names of all the functions in them, as well as the file 
names), that you would prefer?

     2.  The other problem, is that most of the "server" functions 
described above, end by calling complete_pdu.   Complete_pdu (which is 
defined in virtio-9p.c) does many things that are generic, and also a 
few virito specific operations (pushing to the virtqueue, etc.).
     Again, I can use a similar mechanism to the above.  Or is there 
some other way you'd prefer? I'm trying to find a way that has the least 
impact on virtio/qemu maintainers.

Thank you for your time, and any help/insights you can provide.

Sincerely,

Linda Jacobson

Re: [Qemu-devel] virtio-9p

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 3723 bytes --]

On Fri, Aug 07, 2015 at 10:21:47AM -0600, Linda wrote:
>     As background, for the backend, I have been looking at the code, written
> by Anthony Liguori, and maintained by Aneesh Kumar (who I sent this email
> to, earlier).  It looks to me (please correct me if I'm wrong, on this or
> any other point, below) as if Anthony wrote not just a backend transport
> layer, but the server as well.  AFAICT, there is no other Linux 9p server.

There are other Linux 9P servers.  At least there is diod:
https://github.com/chaos/diod

Anthony Liguori didn't write all of the virtio-9p code in QEMU.  Aneesh
Kumar, JV Rao, M. Mohan Kumar, and Harsh Prateek Bora did a lot of the
9P server development in QEMU.

Take a look at git shortlog -nse hw/9pfs

>     virtio-9p.c contains a lot of this server code, the rest spread between
> 13 other files which handle all file access operations, converting them from
> 9p to Linux file system calls.
>     virtio-9p.c also contains some virtio-specific code (although most of
> that is in virtio-device.c).
> 
> The problems I am encountering are the following:
> 
> 1.  In the virio-9p.h is a struct V9fsPDU that contains an element (in the
> middle of the struct) of type VirtQueueElement. Every 9p I/O command
> handler, as well as co-routines and support functions that go with them
> (i.e., a large part of the server), passes a parameter that is a struct
> V9fsPDU.   Almost all of these use only the variable that defines state
> information, and never touch the VirtQueueElement member.
>     The easiest fix for this is to have a separate header file with a
> #define GENERIC_9P_SERVER
>     Then I could modify the virtio-9p.h with:
>             #ifdef GENERIC_9P_SERVER
>                    a union of a void *, a char * (what I use), and a
> VirtQueueElement (guaranteeing the size is unchanged)
>             #else
>                     VirtQueueElement    elem;
>             #endif
> 
>     It's not my favorite construct, but it would involve the least amount of
> changes to the code.   Before I modify a header file, that code, I'm not
> touching, is dependent on, I wanted to know if this is an OK way.  If not,
> is there another way (short of copying fourteen files, and changing the
> names of all the functions in them, as well as the file names), that you
> would prefer?

What is the goal of your project?

If you just want a Linux 9P server, use diod.  You might be able to find
other servers that suit your needs better too (e.g. programming
language, features, etc).

An #ifdef is ugly and if you are going to submit code upstream then a
cleaner solution should be used.  Either separate a VirtIO9fsPDU struct
that contains the generic 9pfsPDU as a field (so that container_of() can
be used to go from 9pfsPDU back to VirtIO9fsPDU).  Or add a void* into
the generic 9pfsPDU so transports can correlate the generic struct with
a transport-specific struct.

>     2.  The other problem, is that most of the "server" functions described
> above, end by calling complete_pdu.   Complete_pdu (which is defined in
> virtio-9p.c) does many things that are generic, and also a few virito
> specific operations (pushing to the virtqueue, etc.).
>     Again, I can use a similar mechanism to the above.  Or is there some
> other way you'd prefer? I'm trying to find a way that has the least impact
> on virtio/qemu maintainers.

The generic PDU struct could have a .complete() function pointer.  This
is how the SCSI subsystem works, for example.  scsi_req_complete() calls
req->bus->info->complete(req, req->status, req->resid) so that the
bus-specific completion behavior is invoked.

Stefan

[-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] virtio-9p

[Linda]

On 8/10/2015 4:10 AM, Stefan Hajnoczi wrote:
> On Fri, Aug 07, 2015 at 10:21:47AM -0600, Linda wrote:
>>      As background, for the backend, I have been looking at the code, 
>> written
>> by Anthony Liguori, and maintained by Aneesh Kumar (who I sent this 
>> email
>> to, earlier).  It looks to me (please correct me if I'm wrong, on 
>> this or
>> any other point, below) as if Anthony wrote not just a backend transport
>> layer, but the server as well.  AFAICT, there is no other Linux 9p 
>> server.
> There are other Linux 9P servers.  At least there is diod:
> https://github.com/chaos/diod
Thank you.  I will look into that.
> Anthony Liguori didn't write all of the virtio-9p code in QEMU.  Aneesh
> Kumar, JV Rao, M. Mohan Kumar, and Harsh Prateek Bora did a lot of the
> 9P server development in QEMU.
>
> Take a look at git shortlog -nse hw/9pfs
>
>>      virtio-9p.c contains a lot of this server code, the rest spread 
>> between
>> 13 other files which handle all file access operations, converting 
>> them from
>> 9p to Linux file system calls.
>>      virtio-9p.c also contains some virtio-specific code (although 
>> most of
>> that is in virtio-device.c).
>>
>> The problems I am encountering are the following:
>>
>> 1.  In the virio-9p.h is a struct V9fsPDU that contains an element 
>> (in the
>> middle of the struct) of type VirtQueueElement. Every 9p I/O command
>> handler, as well as co-routines and support functions that go with them
>> (i.e., a large part of the server), passes a parameter that is a struct
>> V9fsPDU.   Almost all of these use only the variable that defines state
>> information, and never touch the VirtQueueElement member.
>>      The easiest fix for this is to have a separate header file with a
>> #define GENERIC_9P_SERVER
>>      Then I could modify the virtio-9p.h with:
>>              #ifdef GENERIC_9P_SERVER
>>                     a union of a void *, a char * (what I use), and a
>> VirtQueueElement (guaranteeing the size is unchanged)
>>              #else
>>                      VirtQueueElement    elem;
>>              #endif
>>
>>      It's not my favorite construct, but it would involve the least 
>> amount of
>> changes to the code.   Before I modify a header file, that code, I'm not
>> touching, is dependent on, I wanted to know if this is an OK way.  If 
>> not,
>> is there another way (short of copying fourteen files, and changing the
>> names of all the functions in them, as well as the file names), that you
>> would prefer?
> What is the goal of your project?
>
> If you just want a Linux 9P server, use diod.  You might be able to find
> other servers that suit your needs better too (e.g. programming
> language, features, etc).
>
> An #ifdef is ugly and if you are going to submit code upstream then a
> cleaner solution should be used.  Either separate a VirtIO9fsPDU struct
> that contains the generic 9pfsPDU as a field (so that container_of() can
> be used to go from 9pfsPDU back to VirtIO9fsPDU).  Or add a void* into
> the generic 9pfsPDU so transports can correlate the generic struct with
> a transport-specific struct.
I agree about ifdefs being ugly.  I guess I was just trying to save 
space - all I did was add a void * and a pointer to a function
>
>>      2.  The other problem, is that most of the "server" functions 
>> described
>> above, end by calling complete_pdu.   Complete_pdu (which is defined in
>> virtio-9p.c) does many things that are generic, and also a few virito
>> specific operations (pushing to the virtqueue, etc.).
>>      Again, I can use a similar mechanism to the above.  Or is there 
>> some
>> other way you'd prefer? I'm trying to find a way that has the least 
>> impact
>> on virtio/qemu maintainers.
> The generic PDU struct could have a .complete() function pointer. This
> is how the SCSI subsystem works, for example.  scsi_req_complete() calls
> req->bus->info->complete(req, req->status, req->resid) so that the
> bus-specific completion behavior is invoked.
It has a function pointer to a function complete that returns a pointer 
to a Coroutine.   But it uses (in dozens of places) a straight call to a 
complete function.

I will look into diod.   If there are any problems with it, I'll take 
your suggestions above.

Thanks.

Linda
>
> Stefan

[Qemu-devel] Virtio-9p

[Pradeep Kiruvale]

[-- Attachment #1: Type: text/plain, Size: 263 bytes --]

Hi All,

Is virtio-9p-pci device only supports the fsdev deices? I am trying to use
-drive option for applying QoS for block device using Virtio-9p-pci device,
but failing to create/add a device other than fsdev. Can you please help me
on this?

Regards,
Pradeep

[-- Attachment #2: Type: text/html, Size: 343 bytes --]

Re: [Qemu-devel] Virtio-9p

[Greg Kurz]

On Wed, 30 Mar 2016 14:10:38 +0200
Pradeep Kiruvale <pradeepkiruvale@gmail.com> wrote:

> Hi All,
> 
> Is virtio-9p-pci device only supports the fsdev deices? I am trying to use
> -drive option for applying QoS for block device using Virtio-9p-pci device,
> but failing to create/add a device other than fsdev. Can you please help me
> on this?
> 
> Regards,
> Pradeep

Hi Pradeep,

Not sure to catch what you want to do but I confirm that virti-9p-pci only supports
fsdev... if you want a block device, why don't you use virtio-blk-pci ?

Cheers.

--
Greg

Re: [Qemu-devel] Virtio-9p

[Pradeep Kiruvale]

[-- Attachment #1: Type: text/plain, Size: 989 bytes --]

Hi Greg,

Thanks for the reply.

Let me put it this way, virtio-blk-pci is used for block IO on the devices
shared between the guest and the host.
Here I want to share the file and have QoS between the guests. So I am
using the Virtio-9p-pci.

Basically I want to have QoS for virtio-9p-pci.

Regards,
Pradeep

On 30 March 2016 at 16:13, Greg Kurz <gkurz@linux.vnet.ibm.com> wrote:

> On Wed, 30 Mar 2016 14:10:38 +0200
> Pradeep Kiruvale <pradeepkiruvale@gmail.com> wrote:
>
> > Hi All,
> >
> > Is virtio-9p-pci device only supports the fsdev deices? I am trying to
> use
> > -drive option for applying QoS for block device using Virtio-9p-pci
> device,
> > but failing to create/add a device other than fsdev. Can you please help
> me
> > on this?
> >
> > Regards,
> > Pradeep
>
> Hi Pradeep,
>
> Not sure to catch what you want to do but I confirm that virti-9p-pci only
> supports
> fsdev... if you want a block device, why don't you use virtio-blk-pci ?
>
> Cheers.
>
> --
> Greg
>
>

[-- Attachment #2: Type: text/html, Size: 1601 bytes --]

Re: [Qemu-devel] Virtio-9p

[Greg Kurz]

On Wed, 30 Mar 2016 16:27:48 +0200
Pradeep Kiruvale <pradeepkiruvale@gmail.com> wrote:

> Hi Greg,
> 

Hi Pradeep,

> Thanks for the reply.
> 
> Let me put it this way, virtio-blk-pci is used for block IO on the devices
> shared between the guest and the host.

I don't really understand the "devices shared between the guest and the
host" wording... virtio-blk-pci exposes a virtio-blk device through PCI
to the guest. The virtio-blk device can be backed by a file or a block
device from the host.

> Here I want to share the file and have QoS between the guests. So I am
> using the Virtio-9p-pci.
> 

What file ?

> Basically I want to have QoS for virtio-9p-pci.
> 

Can you provide a more detailed scenario on the result you want to reach ?

> Regards,
> Pradeep
> 

Cheers.

--
Greg

> On 30 March 2016 at 16:13, Greg Kurz <gkurz@linux.vnet.ibm.com> wrote:
> 
> > On Wed, 30 Mar 2016 14:10:38 +0200
> > Pradeep Kiruvale <pradeepkiruvale@gmail.com> wrote:
> >
> > > Hi All,
> > >
> > > Is virtio-9p-pci device only supports the fsdev deices? I am trying to
> > use
> > > -drive option for applying QoS for block device using Virtio-9p-pci
> > device,
> > > but failing to create/add a device other than fsdev. Can you please help
> > me
> > > on this?
> > >
> > > Regards,
> > > Pradeep
> >
> > Hi Pradeep,
> >
> > Not sure to catch what you want to do but I confirm that virti-9p-pci only
> > supports
> > fsdev... if you want a block device, why don't you use virtio-blk-pci ?
> >
> > Cheers.
> >
> > --
> > Greg
> >
> >

Re: [Qemu-devel] Virtio-9p

[Greg Kurz]

On Fri, 1 Apr 2016 09:09:32 +0200
Pradeep Kiruvale <pradeepkiruvale@gmail.com> wrote:

> Hi Greg,
> 

Hi Pradeep,

I'm Cc'ing qemu-devel like in your previous posts, so QEMU experts may jump in.

> What I understand from the requirement for our project is if we use
> virtio-blk it caches the pages in the guest. We would like to avoid that

AFAIK this is true when you pass cache=none to -drive on the QEMU command
line. But there are other options such as writeback or writethrough, which
rely on the host page cache.

> and also we want to share those pages across multiple guests and when they
> need some data they should get it from the host instead of using the cached
> data at each guest.
> 

So you want all the guests to attach to the same block device or backing file in
the host, correct ? AFAIK we cannot do that with virtio-blk indeed... and virtio-9p
is only about sharing files, not block devices.

Maybe you could share a big file between the host and all guests with 9p, and
each guest can use the loop device to access the file as a block device... but
even then, you'd have to deal with concurrent accesses...

> Basically we are trying to cut down the memory foot print of the guests.
> 

If you're using KVM and your guests run the same distro or application,
you may try to use KSM (Kernel Same-page Merging) in the host.

> Regards,
> Pradeep
> 
> On 31 March 2016 at 18:12, Greg Kurz <gkurz@linux.vnet.ibm.com> wrote:
> 
> > On Wed, 30 Mar 2016 16:27:48 +0200
> > Pradeep Kiruvale <pradeepkiruvale@gmail.com> wrote:
> >
> > > Hi Greg,
> > >
> >
> > Hi Pradeep,
> >
> > > Thanks for the reply.
> > >
> > > Let me put it this way, virtio-blk-pci is used for block IO on the
> > devices
> > > shared between the guest and the host.
> >
> > I don't really understand the "devices shared between the guest and the
> > host" wording... virtio-blk-pci exposes a virtio-blk device through PCI
> > to the guest. The virtio-blk device can be backed by a file or a block
> > device from the host.
> >
> > > Here I want to share the file and have QoS between the guests. So I am
> > > using the Virtio-9p-pci.
> > >
> >
> > What file ?
> >
> > > Basically I want to have QoS for virtio-9p-pci.
> > >
> >
> > Can you provide a more detailed scenario on the result you want to reach ?
> >
> > > Regards,
> > > Pradeep
> > >
> >
> > Cheers.
> >
> > --
> > Greg
> >
> > > On 30 March 2016 at 16:13, Greg Kurz <gkurz@linux.vnet.ibm.com> wrote:
> > >
> > > > On Wed, 30 Mar 2016 14:10:38 +0200
> > > > Pradeep Kiruvale <pradeepkiruvale@gmail.com> wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > Is virtio-9p-pci device only supports the fsdev deices? I am trying
> > to
> > > > use
> > > > > -drive option for applying QoS for block device using Virtio-9p-pci
> > > > device,
> > > > > but failing to create/add a device other than fsdev. Can you please
> > help
> > > > me
> > > > > on this?
> > > > >
> > > > > Regards,
> > > > > Pradeep
> > > >
> > > > Hi Pradeep,
> > > >
> > > > Not sure to catch what you want to do but I confirm that virti-9p-pci
> > only
> > > > supports
> > > > fsdev... if you want a block device, why don't you use virtio-blk-pci ?
> > > >
> > > > Cheers.
> > > >
> > > > --
> > > > Greg
> > > >
> > > >
> >
> >

Re: [Qemu-devel] Virtio-9p

[Pradeep Kiruvale]

[-- Attachment #1: Type: text/plain, Size: 3606 bytes --]

 Hi Greg,

Thanks for your reply.




>
> I'm Cc'ing qemu-devel like in your previous posts, so QEMU experts may
> jump in.
>
> > What I understand from the requirement for our project is if we use
> > virtio-blk it caches the pages in the guest. We would like to avoid that
>
> AFAIK this is true when you pass cache=none to -drive on the QEMU command
> line. But there are other options such as writeback or writethrough, which
> rely on the host page cache.
>
> Yes, we did explore these options already.


> > and also we want to share those pages across multiple guests and when
> they
> > need some data they should get it from the host instead of using the
> cached
> > data at each guest.
> >
>
> So you want all the guests to attach to the same block device or backing
> file in
> the host, correct ? AFAIK we cannot do that with virtio-blk indeed... and
> virtio-9p
> is only about sharing files, not block devices.
>
>
Yes, we want to share the files.


> Maybe you could share a big file between the host and all guests with 9p,
> and
> each guest can use the loop device to access the file as a block device...
> but
> even then, you'd have to deal with concurrent accesses...
>
> > Basically we are trying to cut down the memory foot print of the guests.
> >
>
> If you're using KVM and your guests run the same distro or application,
> you may try to use KSM (Kernel Same-page Merging) in the host.
>
> We are using these things also, we want to reduce the foot print as much
as possible.

Regards,
Pradeep


> > Regards,
> > Pradeep
> >
> > On 31 March 2016 at 18:12, Greg Kurz <gkurz@linux.vnet.ibm.com> wrote:
> >
> > > On Wed, 30 Mar 2016 16:27:48 +0200
> > > Pradeep Kiruvale <pradeepkiruvale@gmail.com> wrote:
> > >
> > > > Hi Greg,
> > > >
> > >
> > > Hi Pradeep,
> > >
> > > > Thanks for the reply.
> > > >
> > > > Let me put it this way, virtio-blk-pci is used for block IO on the
> > > devices
> > > > shared between the guest and the host.
> > >
> > > I don't really understand the "devices shared between the guest and the
> > > host" wording... virtio-blk-pci exposes a virtio-blk device through PCI
> > > to the guest. The virtio-blk device can be backed by a file or a block
> > > device from the host.
> > >
> > > > Here I want to share the file and have QoS between the guests. So I
> am
> > > > using the Virtio-9p-pci.
> > > >
> > >
> > > What file ?
> > >
> > > > Basically I want to have QoS for virtio-9p-pci.
> > > >
> > >
> > > Can you provide a more detailed scenario on the result you want to
> reach ?
> > >
> > > > Regards,
> > > > Pradeep
> > > >
> > >
> > > Cheers.
> > >
> > > --
> > > Greg
> > >
> > > > On 30 March 2016 at 16:13, Greg Kurz <gkurz@linux.vnet.ibm.com>
> wrote:
> > > >
> > > > > On Wed, 30 Mar 2016 14:10:38 +0200
> > > > > Pradeep Kiruvale <pradeepkiruvale@gmail.com> wrote:
> > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > Is virtio-9p-pci device only supports the fsdev deices? I am
> trying
> > > to
> > > > > use
> > > > > > -drive option for applying QoS for block device using
> Virtio-9p-pci
> > > > > device,
> > > > > > but failing to create/add a device other than fsdev. Can you
> please
> > > help
> > > > > me
> > > > > > on this?
> > > > > >
> > > > > > Regards,
> > > > > > Pradeep
> > > > >
> > > > > Hi Pradeep,
> > > > >
> > > > > Not sure to catch what you want to do but I confirm that
> virti-9p-pci
> > > only
> > > > > supports
> > > > > fsdev... if you want a block device, why don't you use
> virtio-blk-pci ?
> > > > >
> > > > > Cheers.
> > > > >
> > > > > --
> > > > > Greg
> > > > >
> > > > >
> > >
> > >
>
>

[-- Attachment #2: Type: text/html, Size: 5835 bytes --]