[Qemu-devel] [GSoC 2010] Pass-through filesystem support.

[Qemu-devel] [GSoC 2010] Pass-through filesystem support.

[Mohammed Gamal]

Hi,
Now that Cam is almost done with his ivshmem patches, I was thinking
of another idea for GSoC which is improving the pass-though
filesystems.
I've got some questions on that:

1- What does the community prefer to use and improve? CIFS, 9p, or
both? And which is better taken up for GSoC.

2- With respect to CIFS. I wonder how the shares are supposed to be
exposed to the guest. Should the Samba server be modified to be able
to use unix domain sockets instead of TCP ports and then QEMU
communicating on these sockets. With that approach, how should the
guest be able to see the exposed share? And what is the problem of
using Samba with TCP ports?

3- In addition, I see the idea mentions that some Windows code needs
to be written to use network shares on a special interface. What's
that interface? And what's the nature of that Windows code? (a driver
a la "guest additions"?)

Regards,
Mohammed

[Qemu-devel] Re: [GSoC 2010] Pass-through filesystem support.

[Mohammed Gamal]

On Thu, Apr 8, 2010 at 6:01 PM, Mohammed Gamal <m.gamal005@gmail.com> wrote:
> Hi,
> Now that Cam is almost done with his ivshmem patches, I was thinking
> of another idea for GSoC which is improving the pass-though
> filesystems.
> I've got some questions on that:
>
> 1- What does the community prefer to use and improve? CIFS, 9p, or
> both? And which is better taken up for GSoC.
>
> 2- With respect to CIFS. I wonder how the shares are supposed to be
> exposed to the guest. Should the Samba server be modified to be able
> to use unix domain sockets instead of TCP ports and then QEMU
> communicating on these sockets. With that approach, how should the
> guest be able to see the exposed share? And what is the problem of
> using Samba with TCP ports?
>
> 3- In addition, I see the idea mentions that some Windows code needs
> to be written to use network shares on a special interface. What's
> that interface? And what's the nature of that Windows code? (a driver
> a la "guest additions"?)
>
> Regards,
> Mohammed
>

P.S.: A gentle reminder. The proposal submission deadline is tomorrow,
so I'd appreciate responses as soon as possible.

Regards,
Mohammed

[Qemu-devel] Re: [GSoC 2010] Pass-through filesystem support.

[Stefan Hajnoczi]

On Thu, Apr 8, 2010 at 5:02 PM, Mohammed Gamal <m.gamal005@gmail.com> wrote:
> On Thu, Apr 8, 2010 at 6:01 PM, Mohammed Gamal <m.gamal005@gmail.com> wrote:
>> 1- What does the community prefer to use and improve? CIFS, 9p, or
>> both? And which is better taken up for GSoC.

There have been recent patches for filesystem passthrough using 9P:

http://www.mail-archive.com/qemu-devel@nongnu.org/msg28100.html

You might want to consider them if you haven't seen them already.

Stefan

Re: [Qemu-devel] [GSoC 2010] Pass-through filesystem support.

[Luiz Capitulino]

On Thu, 8 Apr 2010 18:01:01 +0200
Mohammed Gamal <m.gamal005@gmail.com> wrote:

> Hi,
> Now that Cam is almost done with his ivshmem patches, I was thinking
> of another idea for GSoC which is improving the pass-though
> filesystems.
> I've got some questions on that:
> 
> 1- What does the community prefer to use and improve? CIFS, 9p, or
> both? And which is better taken up for GSoC.
> 
> 2- With respect to CIFS. I wonder how the shares are supposed to be
> exposed to the guest. Should the Samba server be modified to be able
> to use unix domain sockets instead of TCP ports and then QEMU
> communicating on these sockets. With that approach, how should the
> guest be able to see the exposed share? And what is the problem of
> using Samba with TCP ports?
> 
> 3- In addition, I see the idea mentions that some Windows code needs
> to be written to use network shares on a special interface. What's
> that interface? And what's the nature of that Windows code? (a driver
> a la "guest additions"?)

 CC'ing Aneesh as he's working on that.

Re: [Qemu-devel] [GSoC 2010] Pass-through filesystem support.

[jvrao]

Luiz Capitulino wrote:
> On Thu, 8 Apr 2010 18:01:01 +0200
> Mohammed Gamal <m.gamal005@gmail.com> wrote:
> 
>> Hi,
>> Now that Cam is almost done with his ivshmem patches, I was thinking
>> of another idea for GSoC which is improving the pass-though
>> filesystems.
>> I've got some questions on that:
>>
>> 1- What does the community prefer to use and improve? CIFS, 9p, or
>> both? And which is better taken up for GSoC.

Please look at our recent set of patches. 
We are developing a 9P server for QEMU and client is already part of mainline Linux.
Our goal is to optimize it for virualization environment and will work as FS pass-through
mechanism between host and the guest.

Here is the latest set of patches..

http://www.mail-archive.com/qemu-devel@nongnu.org/msg29267.html

Please let us know if you are interested ... we can coordinate.

Thanks,
JV

>>
>> 2- With respect to CIFS. I wonder how the shares are supposed to be
>> exposed to the guest. Should the Samba server be modified to be able
>> to use unix domain sockets instead of TCP ports and then QEMU
>> communicating on these sockets. With that approach, how should the
>> guest be able to see the exposed share? And what is the problem of
>> using Samba with TCP ports?
>>
>> 3- In addition, I see the idea mentions that some Windows code needs
>> to be written to use network shares on a special interface. What's
>> that interface? And what's the nature of that Windows code? (a driver
>> a la "guest additions"?)
> 
>  CC'ing Aneesh as he's working on that.
> 
> 

Re: [Qemu-devel] [GSoC 2010] Pass-through filesystem support.

[Mohammed Gamal]

On Fri, Apr 9, 2010 at 7:11 PM, jvrao <jvrao@linux.vnet.ibm.com> wrote:
> Luiz Capitulino wrote:
>> On Thu, 8 Apr 2010 18:01:01 +0200
>> Mohammed Gamal <m.gamal005@gmail.com> wrote:
>>
>>> Hi,
>>> Now that Cam is almost done with his ivshmem patches, I was thinking
>>> of another idea for GSoC which is improving the pass-though
>>> filesystems.
>>> I've got some questions on that:
>>>
>>> 1- What does the community prefer to use and improve? CIFS, 9p, or
>>> both? And which is better taken up for GSoC.
>
> Please look at our recent set of patches.
> We are developing a 9P server for QEMU and client is already part of mainline Linux.
> Our goal is to optimize it for virualization environment and will work as FS pass-through
> mechanism between host and the guest.
>
> Here is the latest set of patches..
>
> http://www.mail-archive.com/qemu-devel@nongnu.org/msg29267.html
>
> Please let us know if you are interested ... we can coordinate.
>
> Thanks,
> JV
>

I'd be interested indeed.

>>>
>>> 2- With respect to CIFS. I wonder how the shares are supposed to be
>>> exposed to the guest. Should the Samba server be modified to be able
>>> to use unix domain sockets instead of TCP ports and then QEMU
>>> communicating on these sockets. With that approach, how should the
>>> guest be able to see the exposed share? And what is the problem of
>>> using Samba with TCP ports?
>>>
>>> 3- In addition, I see the idea mentions that some Windows code needs
>>> to be written to use network shares on a special interface. What's
>>> that interface? And what's the nature of that Windows code? (a driver
>>> a la "guest additions"?)
>>
>>  CC'ing Aneesh as he's working on that.
>>
>>
>
>
>

Re: [Qemu-devel] [GSoC 2010] Pass-through filesystem support.

[Jamie Lokier]

Mohammed Gamal wrote:
> 2- With respect to CIFS. I wonder how the shares are supposed to be
> exposed to the guest. Should the Samba server be modified to be able
> to use unix domain sockets instead of TCP ports and then QEMU
> communicating on these sockets. With that approach, how should the
> guest be able to see the exposed share? And what is the problem of
> using Samba with TCP ports?

One problem with TCP ports is it only works when the guest's network
is up :) You can't boot from that.  It also makes things fragile or
difficult if the guest work you are doing involves fiddling with the
network settings.

Doing it over virtio-serial would have many benefits.

On the other hand, Samba+TCP+CIFS does have the advantage of working
with virtually all guest OSes, including Linux / BSDs / Windows /
MacOSX / Solaris etc.  9P only works with Linux as far as I know.

I big problem with Samba at the moment is it's not possible to
instantiate multiple instances of Samba any more, and not as a
non-root user.  That's because it contains some hard-coded paths to
directories of run-time state, at least on Debian/Ubuntu hosts where I
have tried and failed to use qemu's smb option, and there is no config
file option to disable that or even change all the paths.

Patching Samba to make per-user instantiations possible again would go
a long way to making it useful for filesystem passthrough.  Patching
it so you can turn off all the fancy features and have it _just_ serve
a filesystem with the most basic necessary authentication would be
even better.

-- Jamie

Re: [Qemu-devel] [GSoC 2010] Pass-through filesystem support.

[Mohammed Gamal]

On Fri, Apr 9, 2010 at 11:22 PM, Jamie Lokier <jamie@shareable.org> wrote:
> Mohammed Gamal wrote:
>> 2- With respect to CIFS. I wonder how the shares are supposed to be
>> exposed to the guest. Should the Samba server be modified to be able
>> to use unix domain sockets instead of TCP ports and then QEMU
>> communicating on these sockets. With that approach, how should the
>> guest be able to see the exposed share? And what is the problem of
>> using Samba with TCP ports?
>
> One problem with TCP ports is it only works when the guest's network
> is up :) You can't boot from that.  It also makes things fragile or
> difficult if the guest work you are doing involves fiddling with the
> network settings.
>
> Doing it over virtio-serial would have many benefits.
>
> On the other hand, Samba+TCP+CIFS does have the advantage of working
> with virtually all guest OSes, including Linux / BSDs / Windows /
> MacOSX / Solaris etc.  9P only works with Linux as far as I know.
>
> I big problem with Samba at the moment is it's not possible to
> instantiate multiple instances of Samba any more, and not as a
> non-root user.  That's because it contains some hard-coded paths to
> directories of run-time state, at least on Debian/Ubuntu hosts where I
> have tried and failed to use qemu's smb option, and there is no config
> file option to disable that or even change all the paths.
>
> Patching Samba to make per-user instantiations possible again would go
> a long way to making it useful for filesystem passthrough.  Patching
> it so you can turn off all the fancy features and have it _just_ serve
> a filesystem with the most basic necessary authentication would be
> even better.
>
> -- Jamie
>

Hi Jamie,

Thanks for your input.

That's all good and well. The question now is which direction would
the community prefer to go. Would everyone be just happy with
virtio-9p passthrough? Would it support multiple OSs (Windows comes to
mind here)? Or would we eventually need to patch Samba for passthrough
filesystems?

Regards,
Mohammed

Re: [Qemu-devel] [GSoC 2010] Pass-through filesystem support.

[Javier Guerra Giraldez]

On Fri, Apr 9, 2010 at 5:17 PM, Mohammed Gamal <m.gamal005@gmail.com> wrote:
> That's all good and well. The question now is which direction would
> the community prefer to go. Would everyone be just happy with
> virtio-9p passthrough? Would it support multiple OSs (Windows comes to
> mind here)? Or would we eventually need to patch Samba for passthrough
> filesystems?

found this:

http://code.google.com/p/ninefs/

it's a BSD-licensed 9p client for windows.... i have no idea of how
stable / complete / trustable it is; but might be some start


-- 
Javier

Re: [Qemu-devel] [GSoC 2010] Pass-through filesystem support.

[Mohammed Gamal]

On Sat, Apr 10, 2010 at 12:22 AM, Javier Guerra Giraldez
<javier@guerrag.com> wrote:
> On Fri, Apr 9, 2010 at 5:17 PM, Mohammed Gamal <m.gamal005@gmail.com> wrote:
>> That's all good and well. The question now is which direction would
>> the community prefer to go. Would everyone be just happy with
>> virtio-9p passthrough? Would it support multiple OSs (Windows comes to
>> mind here)? Or would we eventually need to patch Samba for passthrough
>> filesystems?
>
> found this:
>
> http://code.google.com/p/ninefs/
>
> it's a BSD-licensed 9p client for windows.... i have no idea of how
> stable / complete / trustable it is; but might be some start
>
>
> --
> Javier
>

Hi Javier,
Thanks for the link. However, I'm still concerned with
interoperability with other operating systems, including non-Windows
ones. I am not sure of how many operating systems actually support 9p,
but I'm almost certain that CIFS would be more widely-supported.
I am still a newbie as far as all this is concerned, so if anyone has
any arguments as to whether which approach should be taken, I'd be
enlightened to hear them.

Regards,
Mohammed

Re: [Qemu-devel] [GSoC 2010] Pass-through filesystem support.

[Jamie Lokier]

Mohammed Gamal wrote:
> Hi Javier,
> Thanks for the link. However, I'm still concerned with
> interoperability with other operating systems, including non-Windows
> ones. I am not sure of how many operating systems actually support 9p,
> but I'm almost certain that CIFS would be more widely-supported.
> I am still a newbie as far as all this is concerned, so if anyone has
> any arguments as to whether which approach should be taken, I'd be
> enlightened to hear them.

To throw a spanner in, the most widely supported filesystem across
operating systems is probably NFS, version 2 :-)

-- Jamie

Re: [Qemu-devel] [GSoC 2010] Pass-through filesystem support.

[Mohammed Gamal]

On Sat, Apr 10, 2010 at 2:12 PM, Jamie Lokier <jamie@shareable.org> wrote:
> Mohammed Gamal wrote:
>> Hi Javier,
>> Thanks for the link. However, I'm still concerned with
>> interoperability with other operating systems, including non-Windows
>> ones. I am not sure of how many operating systems actually support 9p,
>> but I'm almost certain that CIFS would be more widely-supported.
>> I am still a newbie as far as all this is concerned, so if anyone has
>> any arguments as to whether which approach should be taken, I'd be
>> enlightened to hear them.
>
> To throw a spanner in, the most widely supported filesystem across
> operating systems is probably NFS, version 2 :-)
>
> -- Jamie
>

Remember that Windows usage on a VM is not some rare use case, and
it'd be a little bit of a pain from a user's perspective to have to
install a third party NFS client for every VM they use. Having
something supported on the VM out of the box is a better option IMO.

Regards,
Mohammed

Re: [Qemu-devel] [GSoC 2010] Pass-through filesystem support.

[Javier Guerra Giraldez]

On Sat, Apr 10, 2010 at 7:42 AM, Mohammed Gamal <m.gamal005@gmail.com> wrote:
> On Sat, Apr 10, 2010 at 2:12 PM, Jamie Lokier <jamie@shareable.org> wrote:
>> To throw a spanner in, the most widely supported filesystem across
>> operating systems is probably NFS, version 2 :-)
>
> Remember that Windows usage on a VM is not some rare use case, and
> it'd be a little bit of a pain from a user's perspective to have to
> install a third party NFS client for every VM they use. Having
> something supported on the VM out of the box is a better option IMO.

i don't think virtio-CIFS has any more support out of the box (on any
system) than virtio-9P.


-- 
Javier

Re: [Qemu-devel] [GSoC 2010] Pass-through filesystem support.

[Jamie Lokier]

Javier Guerra Giraldez wrote:
> On Sat, Apr 10, 2010 at 7:42 AM, Mohammed Gamal <m.gamal005@gmail.com> wrote:
> > On Sat, Apr 10, 2010 at 2:12 PM, Jamie Lokier <jamie@shareable.org> wrote:
> >> To throw a spanner in, the most widely supported filesystem across
> >> operating systems is probably NFS, version 2 :-)
> >
> > Remember that Windows usage on a VM is not some rare use case, and
> > it'd be a little bit of a pain from a user's perspective to have to
> > install a third party NFS client for every VM they use. Having
> > something supported on the VM out of the box is a better option IMO.
> 
> i don't think virtio-CIFS has any more support out of the box (on any
> system) than virtio-9P.

It doesn't, but at least network-CIFS tends to work ok and is the
method of choice for Windows VMs - when you can setup Samba on the
host (which as previously noted you cannot always do non-disruptively
with current Sambas).

-- Jamie

Re: [Qemu-devel] [GSoC 2010] Pass-through filesystem support.

[Mohammed Gamal]

On Mon, Apr 12, 2010 at 12:29 AM, Jamie Lokier <jamie@shareable.org> wrote:
> Javier Guerra Giraldez wrote:
>> On Sat, Apr 10, 2010 at 7:42 AM, Mohammed Gamal <m.gamal005@gmail.com> wrote:
>> > On Sat, Apr 10, 2010 at 2:12 PM, Jamie Lokier <jamie@shareable.org> wrote:
>> >> To throw a spanner in, the most widely supported filesystem across
>> >> operating systems is probably NFS, version 2 :-)
>> >
>> > Remember that Windows usage on a VM is not some rare use case, and
>> > it'd be a little bit of a pain from a user's perspective to have to
>> > install a third party NFS client for every VM they use. Having
>> > something supported on the VM out of the box is a better option IMO.
>>
>> i don't think virtio-CIFS has any more support out of the box (on any
>> system) than virtio-9P.
>
> It doesn't, but at least network-CIFS tends to work ok and is the
> method of choice for Windows VMs - when you can setup Samba on the
> host (which as previously noted you cannot always do non-disruptively
> with current Sambas).
>
> -- Jamie
>

I think having support for both 9p and CIFS would be the best option.
In that case the user will have the option to use either one,
depending on how their guests support these filesystems. In that case
I'd prefer to work on CIFS support while the 9p effort can still go
on. I don't think both efforts are mutually exclusive.

What do the rest of you guys think?

Regards,
Mohammed

Re: [Qemu-devel] [GSoC 2010] Pass-through filesystem support.

[Alexander Graf]

On 12.04.2010, at 10:15, Mohammed Gamal wrote:

> On Mon, Apr 12, 2010 at 12:29 AM, Jamie Lokier <jamie@shareable.org> wrote:
>> Javier Guerra Giraldez wrote:
>>> On Sat, Apr 10, 2010 at 7:42 AM, Mohammed Gamal <m.gamal005@gmail.com> wrote:
>>>> On Sat, Apr 10, 2010 at 2:12 PM, Jamie Lokier <jamie@shareable.org> wrote:
>>>>> To throw a spanner in, the most widely supported filesystem across
>>>>> operating systems is probably NFS, version 2 :-)
>>>> 
>>>> Remember that Windows usage on a VM is not some rare use case, and
>>>> it'd be a little bit of a pain from a user's perspective to have to
>>>> install a third party NFS client for every VM they use. Having
>>>> something supported on the VM out of the box is a better option IMO.
>>> 
>>> i don't think virtio-CIFS has any more support out of the box (on any
>>> system) than virtio-9P.
>> 
>> It doesn't, but at least network-CIFS tends to work ok and is the
>> method of choice for Windows VMs - when you can setup Samba on the
>> host (which as previously noted you cannot always do non-disruptively
>> with current Sambas).
>> 
>> -- Jamie
>> 
> 
> I think having support for both 9p and CIFS would be the best option.
> In that case the user will have the option to use either one,
> depending on how their guests support these filesystems. In that case
> I'd prefer to work on CIFS support while the 9p effort can still go
> on. I don't think both efforts are mutually exclusive.

I think that the 9p support is handled by the IBM guys pretty well already. They do need help, but I doubt they need as much help as a full GSoC student. You're definitely free to help them out regardless of GSoC :-).

As far as CIFS goes - we currently do use CIFS in the SUSE Studio infrastructure as data interface between host and build guests. About 80% of all backend bugs resulted from that decision. It just breaks in the most ridiculous ways. Just a few examples from our commit log:

  - loopback mounts on CIFS break after heavy I/O
  - moving symlinks fails
  - does not allow ':' in file names

So we're currently waiting for 9p eagerly, as that at least gives us easy control over the stack. And it's a lot less complicated.

Also since -net user does support samba exporting already, I don't see that much value in adding it. I'd rather like to see a virtio-9p driver for Windows.


Alex

Re: [Qemu-devel] [GSoC 2010] Pass-through filesystem support.

[Jamie Lokier]

Alexander Graf wrote:
> Also since -net user does support samba exporting already,

This I'm interested in.  Last time I tried to use it, the "smb="
option didn't work because Samba refused to run when launched with
qemu's mini config file and launched as a regular user.  It needed
access to various hard-coded root-owned directories, and spewed lots
of errors about it into its logs.

On closer inspection, those directories were hard-coded and could not
be changed by the config file, nor could the features they were for be
disabled.

Even if I gave it permission to write to those, by running kvm/qemu as
root, there was plenty of reason to worry that each instance of
qemu-spawned Samba may interfere with the others and with the host's
own....  starting with errors spewed into log files by both Sambas.

So I had to give up on -net user,smb= completely :-(

Is this something that you at SuSE have fixed or simply never encountered?

My problems were with Debian and Ubuntu installations.  I suspect they
might have "fixed" some Samba problems by patching in different problems.

-- Jamie

Re: [Qemu-devel] [GSoC 2010] Pass-through filesystem support.

[Alexander Graf]

On 12.04.2010, at 14:04, Jamie Lokier wrote:

> Alexander Graf wrote:
>> Also since -net user does support samba exporting already,
> 
> This I'm interested in.  Last time I tried to use it, the "smb="
> option didn't work because Samba refused to run when launched with
> qemu's mini config file and launched as a regular user.  It needed
> access to various hard-coded root-owned directories, and spewed lots
> of errors about it into its logs.
> 
> On closer inspection, those directories were hard-coded and could not
> be changed by the config file, nor could the features they were for be
> disabled.
> 
> Even if I gave it permission to write to those, by running kvm/qemu as
> root, there was plenty of reason to worry that each instance of
> qemu-spawned Samba may interfere with the others and with the host's
> own....  starting with errors spewed into log files by both Sambas.
> 
> So I had to give up on -net user,smb= completely :-(
> 
> Is this something that you at SuSE have fixed or simply never encountered?

I think I tried -net user,smb some years ago and found it not working either.

Either way, this sounds more like someone should spend a few hours getting samba fixed instead of occupying a talented student for a full summer where he could instead to more productive things :-).


Alex

Re: [Qemu-devel] [GSoC 2010] Pass-through filesystem support.

[Jamie Lokier]

Mohammed Gamal wrote:
> On Mon, Apr 12, 2010 at 12:29 AM, Jamie Lokier <jamie@shareable.org> wrote:
> > Javier Guerra Giraldez wrote:
> >> On Sat, Apr 10, 2010 at 7:42 AM, Mohammed Gamal <m.gamal005@gmail.com> wrote:
> >> > On Sat, Apr 10, 2010 at 2:12 PM, Jamie Lokier <jamie@shareable.org> wrote:
> >> >> To throw a spanner in, the most widely supported filesystem across
> >> >> operating systems is probably NFS, version 2 :-)
> >> >
> >> > Remember that Windows usage on a VM is not some rare use case, and
> >> > it'd be a little bit of a pain from a user's perspective to have to
> >> > install a third party NFS client for every VM they use. Having
> >> > something supported on the VM out of the box is a better option IMO.
> >>
> >> i don't think virtio-CIFS has any more support out of the box (on any
> >> system) than virtio-9P.
> >
> > It doesn't, but at least network-CIFS tends to work ok and is the
> > method of choice for Windows VMs - when you can setup Samba on the
> > host (which as previously noted you cannot always do non-disruptively
> > with current Sambas).
> >
> > -- Jamie
> >
> 
> I think having support for both 9p and CIFS would be the best option.
> In that case the user will have the option to use either one,
> depending on how their guests support these filesystems. In that case
> I'd prefer to work on CIFS support while the 9p effort can still go
> on. I don't think both efforts are mutually exclusive.
> 
> What do the rest of you guys think?

I only noted NFS because most old OSes do not support CIFS or 9P -
especially all the old unixes.

I don't think old versions of MS-DOS and Windows (95, 98, ME, Nt4?)
even support current CIFS.  They need extra server settings to work
- such as setting passwords on the server to non-encrypted and other quirks.

Meanwhile Windows Vista/2008/7 works better with SMB2, not CIFS, to
properly see symlinks and hard links.

So there is no really nice out of the box file service which works
easily with all guest OSes.

I'm guessing that out of all the filesystems, CIFS is the most widely
supported in recent OSes (released in the last 10 years).  But I'm not
really sure what the state of CIFS is for non-Windows, non-Linux,
non-BSD guests.

I'm not sure why 9P is being pursued.  Does anything much support it,
or do all OSes except quite recent Linux need a custom driver for 9P?
Even Linux only got the first commit in the kernel 5 years ago, so
probably it was only about 3 years ago that it will have begun
appearing in stable distros, if at all.  Filesystem passthrough to
Linux guests installed in the last couple of years is a useful
feature, and I know that for many people that is their only use of
KVM, but compared with CIFS' broad support it seems like quite a
narrow goal.

-- Jamie

Re: [Qemu-devel] [GSoC 2010] Pass-through filesystem support.

[Alexander Graf]

On 12.04.2010, at 13:58, Jamie Lokier wrote:

> Mohammed Gamal wrote:
>> On Mon, Apr 12, 2010 at 12:29 AM, Jamie Lokier <jamie@shareable.org> wrote:
>>> Javier Guerra Giraldez wrote:
>>>> On Sat, Apr 10, 2010 at 7:42 AM, Mohammed Gamal <m.gamal005@gmail.com> wrote:
>>>>> On Sat, Apr 10, 2010 at 2:12 PM, Jamie Lokier <jamie@shareable.org> wrote:
>>>>>> To throw a spanner in, the most widely supported filesystem across
>>>>>> operating systems is probably NFS, version 2 :-)
>>>>> 
>>>>> Remember that Windows usage on a VM is not some rare use case, and
>>>>> it'd be a little bit of a pain from a user's perspective to have to
>>>>> install a third party NFS client for every VM they use. Having
>>>>> something supported on the VM out of the box is a better option IMO.
>>>> 
>>>> i don't think virtio-CIFS has any more support out of the box (on any
>>>> system) than virtio-9P.
>>> 
>>> It doesn't, but at least network-CIFS tends to work ok and is the
>>> method of choice for Windows VMs - when you can setup Samba on the
>>> host (which as previously noted you cannot always do non-disruptively
>>> with current Sambas).
>>> 
>>> -- Jamie
>>> 
>> 
>> I think having support for both 9p and CIFS would be the best option.
>> In that case the user will have the option to use either one,
>> depending on how their guests support these filesystems. In that case
>> I'd prefer to work on CIFS support while the 9p effort can still go
>> on. I don't think both efforts are mutually exclusive.
>> 
>> What do the rest of you guys think?
> 
> I only noted NFS because most old OSes do not support CIFS or 9P -
> especially all the old unixes.
> 
> I don't think old versions of MS-DOS and Windows (95, 98, ME, Nt4?)
> even support current CIFS.  They need extra server settings to work
> - such as setting passwords on the server to non-encrypted and other quirks.
> 
> Meanwhile Windows Vista/2008/7 works better with SMB2, not CIFS, to
> properly see symlinks and hard links.
> 
> So there is no really nice out of the box file service which works
> easily with all guest OSes.
> 
> I'm guessing that out of all the filesystems, CIFS is the most widely
> supported in recent OSes (released in the last 10 years).  But I'm not
> really sure what the state of CIFS is for non-Windows, non-Linux,
> non-BSD guests.

So what? If you want to have direct host fs access, install guest drivers. If you can't, set up networking and use CIFS or NFS or whatever.

> 
> I'm not sure why 9P is being pursued.  Does anything much support it,
> or do all OSes except quite recent Linux need a custom driver for 9P?
> Even Linux only got the first commit in the kernel 5 years ago, so
> probably it was only about 3 years ago that it will have begun
> appearing in stable distros, if at all.  Filesystem passthrough to
> Linux guests installed in the last couple of years is a useful
> feature, and I know that for many people that is their only use of
> KVM, but compared with CIFS' broad support it seems like quite a
> narrow goal.

The goal is to have something simple and fast. We can fine-tune 9P to align with the Linux VFS structures, making it really little overhead (and little headache too). For Windows guests, nothing prevents us to expose yet another 9P flavor. That again would be aligned well with Windows's VFS and be slim and fast there.

The biggest problem I see with CIFS is that it's a huge beast. There are a lot of corner cases where it just doesn't fit in. See my previous mail for more details.


Alex

Re: [Qemu-devel] [GSoC 2010] Pass-through filesystem support.

[jvrao]

Alexander Graf wrote:
> On 12.04.2010, at 13:58, Jamie Lokier wrote:
> 
>> Mohammed Gamal wrote:
>>> On Mon, Apr 12, 2010 at 12:29 AM, Jamie Lokier <jamie@shareable.org> wrote:
>>>> Javier Guerra Giraldez wrote:
>>>>> On Sat, Apr 10, 2010 at 7:42 AM, Mohammed Gamal <m.gamal005@gmail.com> wrote:
>>>>>> On Sat, Apr 10, 2010 at 2:12 PM, Jamie Lokier <jamie@shareable.org> wrote:
>>>>>>> To throw a spanner in, the most widely supported filesystem across
>>>>>>> operating systems is probably NFS, version 2 :-)
>>>>>> Remember that Windows usage on a VM is not some rare use case, and
>>>>>> it'd be a little bit of a pain from a user's perspective to have to
>>>>>> install a third party NFS client for every VM they use. Having
>>>>>> something supported on the VM out of the box is a better option IMO.
>>>>> i don't think virtio-CIFS has any more support out of the box (on any
>>>>> system) than virtio-9P.
>>>> It doesn't, but at least network-CIFS tends to work ok and is the
>>>> method of choice for Windows VMs - when you can setup Samba on the
>>>> host (which as previously noted you cannot always do non-disruptively
>>>> with current Sambas).
>>>>
>>>> -- Jamie
>>>>
>>> I think having support for both 9p and CIFS would be the best option.
>>> In that case the user will have the option to use either one,
>>> depending on how their guests support these filesystems. In that case
>>> I'd prefer to work on CIFS support while the 9p effort can still go
>>> on. I don't think both efforts are mutually exclusive.
>>>
>>> What do the rest of you guys think?
>> I only noted NFS because most old OSes do not support CIFS or 9P -
>> especially all the old unixes.
>>
>> I don't think old versions of MS-DOS and Windows (95, 98, ME, Nt4?)
>> even support current CIFS.  They need extra server settings to work
>> - such as setting passwords on the server to non-encrypted and other quirks.
>>
>> Meanwhile Windows Vista/2008/7 works better with SMB2, not CIFS, to
>> properly see symlinks and hard links.
>>
>> So there is no really nice out of the box file service which works
>> easily with all guest OSes.
>>
>> I'm guessing that out of all the filesystems, CIFS is the most widely
>> supported in recent OSes (released in the last 10 years).  But I'm not
>> really sure what the state of CIFS is for non-Windows, non-Linux,
>> non-BSD guests.
> 
> So what? If you want to have direct host fs access, install guest drivers. If you can't, set up networking and use CIFS or NFS or whatever.
> 
>> I'm not sure why 9P is being pursued.  Does anything much support it,
>> or do all OSes except quite recent Linux need a custom driver for 9P?
>> Even Linux only got the first commit in the kernel 5 years ago, so
>> probably it was only about 3 years ago that it will have begun
>> appearing in stable distros, if at all.  Filesystem passthrough to
>> Linux guests installed in the last couple of years is a useful
>> feature, and I know that for many people that is their only use of
>> KVM, but compared with CIFS' broad support it seems like quite a
>> narrow goal.
> 
> The goal is to have something simple and fast. We can fine-tune 9P to align with the Linux VFS structures, making it really little overhead (and little headache too). For Windows guests, nothing prevents us to expose yet another 9P flavor. That again would be aligned well with Windows's VFS and be slim and fast there.
> 
> The biggest problem I see with CIFS is that it's a huge beast. There are a lot of corner cases where it just doesn't fit in. See my previous mail for more details.
> 
As Alex mentioned, 9P is chosen for its mere simplicity and easy adaptability.
NFS and CIFS does not give that flexibility. As we mentioned in the patch series, we are 
already seeing better numbers with 9P. Looking ahead 9P can embed KVM/QEMU knowledge
to share physical resources like memory/cache between the host and the guest.

I think looking into the windows side of 9P client would be great option too. 
The current patch on the mailing list supports .U (unix) protocol and will be introducing
.L (Linux) variant as we move forward.

- JV


> 
> Alex
> 
> 
> 

Re: [Qemu-devel] [GSoC 2010] Pass-through filesystem support.

[jvrao]

jvrao wrote:
> Alexander Graf wrote:
>> On 12.04.2010, at 13:58, Jamie Lokier wrote:
>>
>>> Mohammed Gamal wrote:
>>>> On Mon, Apr 12, 2010 at 12:29 AM, Jamie Lokier <jamie@shareable.org> wrote:
>>>>> Javier Guerra Giraldez wrote:
>>>>>> On Sat, Apr 10, 2010 at 7:42 AM, Mohammed Gamal <m.gamal005@gmail.com> wrote:
>>>>>>> On Sat, Apr 10, 2010 at 2:12 PM, Jamie Lokier <jamie@shareable.org> wrote:
>>>>>>>> To throw a spanner in, the most widely supported filesystem across
>>>>>>>> operating systems is probably NFS, version 2 :-)
>>>>>>> Remember that Windows usage on a VM is not some rare use case, and
>>>>>>> it'd be a little bit of a pain from a user's perspective to have to
>>>>>>> install a third party NFS client for every VM they use. Having
>>>>>>> something supported on the VM out of the box is a better option IMO.
>>>>>> i don't think virtio-CIFS has any more support out of the box (on any
>>>>>> system) than virtio-9P.
>>>>> It doesn't, but at least network-CIFS tends to work ok and is the
>>>>> method of choice for Windows VMs - when you can setup Samba on the
>>>>> host (which as previously noted you cannot always do non-disruptively
>>>>> with current Sambas).
>>>>>
>>>>> -- Jamie
>>>>>
>>>> I think having support for both 9p and CIFS would be the best option.
>>>> In that case the user will have the option to use either one,
>>>> depending on how their guests support these filesystems. In that case
>>>> I'd prefer to work on CIFS support while the 9p effort can still go
>>>> on. I don't think both efforts are mutually exclusive.
>>>>
>>>> What do the rest of you guys think?
>>> I only noted NFS because most old OSes do not support CIFS or 9P -
>>> especially all the old unixes.
>>>
>>> I don't think old versions of MS-DOS and Windows (95, 98, ME, Nt4?)
>>> even support current CIFS.  They need extra server settings to work
>>> - such as setting passwords on the server to non-encrypted and other quirks.
>>>
>>> Meanwhile Windows Vista/2008/7 works better with SMB2, not CIFS, to
>>> properly see symlinks and hard links.
>>>
>>> So there is no really nice out of the box file service which works
>>> easily with all guest OSes.
>>>
>>> I'm guessing that out of all the filesystems, CIFS is the most widely
>>> supported in recent OSes (released in the last 10 years).  But I'm not
>>> really sure what the state of CIFS is for non-Windows, non-Linux,
>>> non-BSD guests.
>> So what? If you want to have direct host fs access, install guest drivers. If you can't, set up networking and use CIFS or NFS or whatever.
>>
>>> I'm not sure why 9P is being pursued.  Does anything much support it,
>>> or do all OSes except quite recent Linux need a custom driver for 9P?
>>> Even Linux only got the first commit in the kernel 5 years ago, so
>>> probably it was only about 3 years ago that it will have begun
>>> appearing in stable distros, if at all.  Filesystem passthrough to
>>> Linux guests installed in the last couple of years is a useful
>>> feature, and I know that for many people that is their only use of
>>> KVM, but compared with CIFS' broad support it seems like quite a
>>> narrow goal.
>> The goal is to have something simple and fast. We can fine-tune 9P to align with the Linux VFS structures, making it really little overhead (and little headache too). For Windows guests, nothing prevents us to expose yet another 9P flavor. That again would be aligned well with Windows's VFS and be slim and fast there.
>>
>> The biggest problem I see with CIFS is that it's a huge beast. There are a lot of corner cases where it just doesn't fit in. See my previous mail for more details.
>>
> As Alex mentioned, 9P is chosen for its mere simplicity and easy adaptability.
> NFS and CIFS does not give that flexibility. As we mentioned in the patch series, we are 
> already seeing better numbers with 9P. Looking ahead 9P can embed KVM/QEMU knowledge
> to share physical resources like memory/cache between the host and the guest.
> 
> I think looking into the windows side of 9P client would be great option too. 
> The current patch on the mailing list supports .U (unix) protocol and will be introducing
> .L (Linux) variant as we move forward.

Hi Mohammed,
Please let us know once you decide on where your interest lies.
Will be glad to have you on VirtFS (9P) though. :)


- JV

> 
> - JV
> 
> 
>> Alex
>>
>>
>>
> 
> 
> 
> 

Re: [Qemu-devel] [GSoC 2010] Pass-through filesystem support.

[Mohammed Gamal]

On Tue, Apr 13, 2010 at 9:08 PM, jvrao <jvrao@linux.vnet.ibm.com> wrote:
> jvrao wrote:
>> Alexander Graf wrote:
>>> On 12.04.2010, at 13:58, Jamie Lokier wrote:
>>>
>>>> Mohammed Gamal wrote:
>>>>> On Mon, Apr 12, 2010 at 12:29 AM, Jamie Lokier <jamie@shareable.org> wrote:
>>>>>> Javier Guerra Giraldez wrote:
>>>>>>> On Sat, Apr 10, 2010 at 7:42 AM, Mohammed Gamal <m.gamal005@gmail.com> wrote:
>>>>>>>> On Sat, Apr 10, 2010 at 2:12 PM, Jamie Lokier <jamie@shareable.org> wrote:
>>>>>>>>> To throw a spanner in, the most widely supported filesystem across
>>>>>>>>> operating systems is probably NFS, version 2 :-)
>>>>>>>> Remember that Windows usage on a VM is not some rare use case, and
>>>>>>>> it'd be a little bit of a pain from a user's perspective to have to
>>>>>>>> install a third party NFS client for every VM they use. Having
>>>>>>>> something supported on the VM out of the box is a better option IMO.
>>>>>>> i don't think virtio-CIFS has any more support out of the box (on any
>>>>>>> system) than virtio-9P.
>>>>>> It doesn't, but at least network-CIFS tends to work ok and is the
>>>>>> method of choice for Windows VMs - when you can setup Samba on the
>>>>>> host (which as previously noted you cannot always do non-disruptively
>>>>>> with current Sambas).
>>>>>>
>>>>>> -- Jamie
>>>>>>
>>>>> I think having support for both 9p and CIFS would be the best option.
>>>>> In that case the user will have the option to use either one,
>>>>> depending on how their guests support these filesystems. In that case
>>>>> I'd prefer to work on CIFS support while the 9p effort can still go
>>>>> on. I don't think both efforts are mutually exclusive.
>>>>>
>>>>> What do the rest of you guys think?
>>>> I only noted NFS because most old OSes do not support CIFS or 9P -
>>>> especially all the old unixes.
>>>>
>>>> I don't think old versions of MS-DOS and Windows (95, 98, ME, Nt4?)
>>>> even support current CIFS.  They need extra server settings to work
>>>> - such as setting passwords on the server to non-encrypted and other quirks.
>>>>
>>>> Meanwhile Windows Vista/2008/7 works better with SMB2, not CIFS, to
>>>> properly see symlinks and hard links.
>>>>
>>>> So there is no really nice out of the box file service which works
>>>> easily with all guest OSes.
>>>>
>>>> I'm guessing that out of all the filesystems, CIFS is the most widely
>>>> supported in recent OSes (released in the last 10 years).  But I'm not
>>>> really sure what the state of CIFS is for non-Windows, non-Linux,
>>>> non-BSD guests.
>>> So what? If you want to have direct host fs access, install guest drivers. If you can't, set up networking and use CIFS or NFS or whatever.
>>>
>>>> I'm not sure why 9P is being pursued.  Does anything much support it,
>>>> or do all OSes except quite recent Linux need a custom driver for 9P?
>>>> Even Linux only got the first commit in the kernel 5 years ago, so
>>>> probably it was only about 3 years ago that it will have begun
>>>> appearing in stable distros, if at all.  Filesystem passthrough to
>>>> Linux guests installed in the last couple of years is a useful
>>>> feature, and I know that for many people that is their only use of
>>>> KVM, but compared with CIFS' broad support it seems like quite a
>>>> narrow goal.
>>> The goal is to have something simple and fast. We can fine-tune 9P to align with the Linux VFS structures, making it really little overhead (and little headache too). For Windows guests, nothing prevents us to expose yet another 9P flavor. That again would be aligned well with Windows's VFS and be slim and fast there.
>>>
>>> The biggest problem I see with CIFS is that it's a huge beast. There are a lot of corner cases where it just doesn't fit in. See my previous mail for more details.
>>>
>> As Alex mentioned, 9P is chosen for its mere simplicity and easy adaptability.
>> NFS and CIFS does not give that flexibility. As we mentioned in the patch series, we are
>> already seeing better numbers with 9P. Looking ahead 9P can embed KVM/QEMU knowledge
>> to share physical resources like memory/cache between the host and the guest.
>>
>> I think looking into the windows side of 9P client would be great option too.
>> The current patch on the mailing list supports .U (unix) protocol and will be introducing
>> .L (Linux) variant as we move forward.
>
> Hi Mohammed,
> Please let us know once you decide on where your interest lies.
> Will be glad to have you on VirtFS (9P) though. :)
>
>
> - JV
>

It seems the community is more keen on getting 9P support merged than
getting CIFS supported, and they have made good points to support
their argument. I'm not sure whether work on this project could fit in
as a GSoC project and if there is much remaining work that could make
it fit in that direction. But I'd be glad to volunteer anyway :)

Regards,
Mohammed

Re: [Qemu-devel] [GSoC 2010] Pass-through filesystem support.

[jvrao]

Mohammed Gamal wrote:
> On Tue, Apr 13, 2010 at 9:08 PM, jvrao <jvrao@linux.vnet.ibm.com> wrote:
>> jvrao wrote:
>>> Alexander Graf wrote:
>>>> On 12.04.2010, at 13:58, Jamie Lokier wrote:
>>>>
>>>>> Mohammed Gamal wrote:
>>>>>> On Mon, Apr 12, 2010 at 12:29 AM, Jamie Lokier <jamie@shareable.org> wrote:
>>>>>>> Javier Guerra Giraldez wrote:
>>>>>>>> On Sat, Apr 10, 2010 at 7:42 AM, Mohammed Gamal <m.gamal005@gmail.com> wrote:
>>>>>>>>> On Sat, Apr 10, 2010 at 2:12 PM, Jamie Lokier <jamie@shareable.org> wrote:
>>>>>>>>>> To throw a spanner in, the most widely supported filesystem across
>>>>>>>>>> operating systems is probably NFS, version 2 :-)
>>>>>>>>> Remember that Windows usage on a VM is not some rare use case, and
>>>>>>>>> it'd be a little bit of a pain from a user's perspective to have to
>>>>>>>>> install a third party NFS client for every VM they use. Having
>>>>>>>>> something supported on the VM out of the box is a better option IMO.
>>>>>>>> i don't think virtio-CIFS has any more support out of the box (on any
>>>>>>>> system) than virtio-9P.
>>>>>>> It doesn't, but at least network-CIFS tends to work ok and is the
>>>>>>> method of choice for Windows VMs - when you can setup Samba on the
>>>>>>> host (which as previously noted you cannot always do non-disruptively
>>>>>>> with current Sambas).
>>>>>>>
>>>>>>> -- Jamie
>>>>>>>
>>>>>> I think having support for both 9p and CIFS would be the best option.
>>>>>> In that case the user will have the option to use either one,
>>>>>> depending on how their guests support these filesystems. In that case
>>>>>> I'd prefer to work on CIFS support while the 9p effort can still go
>>>>>> on. I don't think both efforts are mutually exclusive.
>>>>>>
>>>>>> What do the rest of you guys think?
>>>>> I only noted NFS because most old OSes do not support CIFS or 9P -
>>>>> especially all the old unixes.
>>>>>
>>>>> I don't think old versions of MS-DOS and Windows (95, 98, ME, Nt4?)
>>>>> even support current CIFS.  They need extra server settings to work
>>>>> - such as setting passwords on the server to non-encrypted and other quirks.
>>>>>
>>>>> Meanwhile Windows Vista/2008/7 works better with SMB2, not CIFS, to
>>>>> properly see symlinks and hard links.
>>>>>
>>>>> So there is no really nice out of the box file service which works
>>>>> easily with all guest OSes.
>>>>>
>>>>> I'm guessing that out of all the filesystems, CIFS is the most widely
>>>>> supported in recent OSes (released in the last 10 years).  But I'm not
>>>>> really sure what the state of CIFS is for non-Windows, non-Linux,
>>>>> non-BSD guests.
>>>> So what? If you want to have direct host fs access, install guest drivers. If you can't, set up networking and use CIFS or NFS or whatever.
>>>>
>>>>> I'm not sure why 9P is being pursued.  Does anything much support it,
>>>>> or do all OSes except quite recent Linux need a custom driver for 9P?
>>>>> Even Linux only got the first commit in the kernel 5 years ago, so
>>>>> probably it was only about 3 years ago that it will have begun
>>>>> appearing in stable distros, if at all.  Filesystem passthrough to
>>>>> Linux guests installed in the last couple of years is a useful
>>>>> feature, and I know that for many people that is their only use of
>>>>> KVM, but compared with CIFS' broad support it seems like quite a
>>>>> narrow goal.
>>>> The goal is to have something simple and fast. We can fine-tune 9P to align with the Linux VFS structures, making it really little overhead (and little headache too). For Windows guests, nothing prevents us to expose yet another 9P flavor. That again would be aligned well with Windows's VFS and be slim and fast there.
>>>>
>>>> The biggest problem I see with CIFS is that it's a huge beast. There are a lot of corner cases where it just doesn't fit in. See my previous mail for more details.
>>>>
>>> As Alex mentioned, 9P is chosen for its mere simplicity and easy adaptability.
>>> NFS and CIFS does not give that flexibility. As we mentioned in the patch series, we are
>>> already seeing better numbers with 9P. Looking ahead 9P can embed KVM/QEMU knowledge
>>> to share physical resources like memory/cache between the host and the guest.
>>>
>>> I think looking into the windows side of 9P client would be great option too.
>>> The current patch on the mailing list supports .U (unix) protocol and will be introducing
>>> .L (Linux) variant as we move forward.
>> Hi Mohammed,
>> Please let us know once you decide on where your interest lies.
>> Will be glad to have you on VirtFS (9P) though. :)
>>
>>
>> - JV
>>
> 
> It seems the community is more keen on getting 9P support merged than
> getting CIFS supported, and they have made good points to support
> their argument. I'm not sure whether work on this project could fit in
> as a GSoC project and if there is much remaining work that could make
> it fit in that direction. But I'd be glad to volunteer anyway :)

I was thinking over the wk-end what fits your schedule and your interest areas. :)

One thing I can think of is, making NFS server export VirtFS mount on the guest to
the external world. This works fine now if we enable loose cache option in 9P mount.
But I think we should identify and make this exports work even otherwise. 
Please let me know if this is something that you want to sign up. :)

Thanks,
JV

> 
> Regards,
> Mohammed
> 
> 

Re: [Qemu-devel] [GSoC 2010] Pass-through filesystem support.

[Mohammed Gamal]

On Tue, Apr 20, 2010 at 12:54 AM, jvrao <jvrao@linux.vnet.ibm.com> wrote:
> Mohammed Gamal wrote:
>> On Tue, Apr 13, 2010 at 9:08 PM, jvrao <jvrao@linux.vnet.ibm.com> wrote:
>>> jvrao wrote:
>>>> Alexander Graf wrote:
>>>>> On 12.04.2010, at 13:58, Jamie Lokier wrote:
>>>>>
>>>>>> Mohammed Gamal wrote:
>>>>>>> On Mon, Apr 12, 2010 at 12:29 AM, Jamie Lokier <jamie@shareable.org> wrote:
>>>>>>>> Javier Guerra Giraldez wrote:
>>>>>>>>> On Sat, Apr 10, 2010 at 7:42 AM, Mohammed Gamal <m.gamal005@gmail.com> wrote:
>>>>>>>>>> On Sat, Apr 10, 2010 at 2:12 PM, Jamie Lokier <jamie@shareable.org> wrote:
>>>>>>>>>>> To throw a spanner in, the most widely supported filesystem across
>>>>>>>>>>> operating systems is probably NFS, version 2 :-)
>>>>>>>>>> Remember that Windows usage on a VM is not some rare use case, and
>>>>>>>>>> it'd be a little bit of a pain from a user's perspective to have to
>>>>>>>>>> install a third party NFS client for every VM they use. Having
>>>>>>>>>> something supported on the VM out of the box is a better option IMO.
>>>>>>>>> i don't think virtio-CIFS has any more support out of the box (on any
>>>>>>>>> system) than virtio-9P.
>>>>>>>> It doesn't, but at least network-CIFS tends to work ok and is the
>>>>>>>> method of choice for Windows VMs - when you can setup Samba on the
>>>>>>>> host (which as previously noted you cannot always do non-disruptively
>>>>>>>> with current Sambas).
>>>>>>>>
>>>>>>>> -- Jamie
>>>>>>>>
>>>>>>> I think having support for both 9p and CIFS would be the best option.
>>>>>>> In that case the user will have the option to use either one,
>>>>>>> depending on how their guests support these filesystems. In that case
>>>>>>> I'd prefer to work on CIFS support while the 9p effort can still go
>>>>>>> on. I don't think both efforts are mutually exclusive.
>>>>>>>
>>>>>>> What do the rest of you guys think?
>>>>>> I only noted NFS because most old OSes do not support CIFS or 9P -
>>>>>> especially all the old unixes.
>>>>>>
>>>>>> I don't think old versions of MS-DOS and Windows (95, 98, ME, Nt4?)
>>>>>> even support current CIFS.  They need extra server settings to work
>>>>>> - such as setting passwords on the server to non-encrypted and other quirks.
>>>>>>
>>>>>> Meanwhile Windows Vista/2008/7 works better with SMB2, not CIFS, to
>>>>>> properly see symlinks and hard links.
>>>>>>
>>>>>> So there is no really nice out of the box file service which works
>>>>>> easily with all guest OSes.
>>>>>>
>>>>>> I'm guessing that out of all the filesystems, CIFS is the most widely
>>>>>> supported in recent OSes (released in the last 10 years).  But I'm not
>>>>>> really sure what the state of CIFS is for non-Windows, non-Linux,
>>>>>> non-BSD guests.
>>>>> So what? If you want to have direct host fs access, install guest drivers. If you can't, set up networking and use CIFS or NFS or whatever.
>>>>>
>>>>>> I'm not sure why 9P is being pursued.  Does anything much support it,
>>>>>> or do all OSes except quite recent Linux need a custom driver for 9P?
>>>>>> Even Linux only got the first commit in the kernel 5 years ago, so
>>>>>> probably it was only about 3 years ago that it will have begun
>>>>>> appearing in stable distros, if at all.  Filesystem passthrough to
>>>>>> Linux guests installed in the last couple of years is a useful
>>>>>> feature, and I know that for many people that is their only use of
>>>>>> KVM, but compared with CIFS' broad support it seems like quite a
>>>>>> narrow goal.
>>>>> The goal is to have something simple and fast. We can fine-tune 9P to align with the Linux VFS structures, making it really little overhead (and little headache too). For Windows guests, nothing prevents us to expose yet another 9P flavor. That again would be aligned well with Windows's VFS and be slim and fast there.
>>>>>
>>>>> The biggest problem I see with CIFS is that it's a huge beast. There are a lot of corner cases where it just doesn't fit in. See my previous mail for more details.
>>>>>
>>>> As Alex mentioned, 9P is chosen for its mere simplicity and easy adaptability.
>>>> NFS and CIFS does not give that flexibility. As we mentioned in the patch series, we are
>>>> already seeing better numbers with 9P. Looking ahead 9P can embed KVM/QEMU knowledge
>>>> to share physical resources like memory/cache between the host and the guest.
>>>>
>>>> I think looking into the windows side of 9P client would be great option too.
>>>> The current patch on the mailing list supports .U (unix) protocol and will be introducing
>>>> .L (Linux) variant as we move forward.
>>> Hi Mohammed,
>>> Please let us know once you decide on where your interest lies.
>>> Will be glad to have you on VirtFS (9P) though. :)
>>>
>>>
>>> - JV
>>>
>>
>> It seems the community is more keen on getting 9P support merged than
>> getting CIFS supported, and they have made good points to support
>> their argument. I'm not sure whether work on this project could fit in
>> as a GSoC project and if there is much remaining work that could make
>> it fit in that direction. But I'd be glad to volunteer anyway :)
>
> I was thinking over the wk-end what fits your schedule and your interest areas. :)
>
> One thing I can think of is, making NFS server export VirtFS mount on the guest to
> the external world. This works fine now if we enable loose cache option in 9P mount.
> But I think we should identify and make this exports work even otherwise.
> Please let me know if this is something that you want to sign up. :)
>
> Thanks,
> JV
>

This'd be something interesting to do. I wonder if that would fit in
the GSoC timeframe, or whether it'd be a little too short. So how long
you'd estimate something like that would take?

Regards,
Mohammed

Re: [Qemu-devel] [GSoC 2010] Pass-through filesystem support.

[jvrao]

Mohammed Gamal wrote:
> On Tue, Apr 20, 2010 at 12:54 AM, jvrao <jvrao@linux.vnet.ibm.com> wrote:
>> Mohammed Gamal wrote:
>>> On Tue, Apr 13, 2010 at 9:08 PM, jvrao <jvrao@linux.vnet.ibm.com> wrote:
>>>> jvrao wrote:
>>>>> Alexander Graf wrote:
>>>>>> On 12.04.2010, at 13:58, Jamie Lokier wrote:
>>>>>>
>>>>>>> Mohammed Gamal wrote:
>>>>>>>> On Mon, Apr 12, 2010 at 12:29 AM, Jamie Lokier <jamie@shareable.org> wrote:
>>>>>>>>> Javier Guerra Giraldez wrote:
>>>>>>>>>> On Sat, Apr 10, 2010 at 7:42 AM, Mohammed Gamal <m.gamal005@gmail.com> wrote:
>>>>>>>>>>> On Sat, Apr 10, 2010 at 2:12 PM, Jamie Lokier <jamie@shareable.org> wrote:
>>>>>>>>>>>> To throw a spanner in, the most widely supported filesystem across
>>>>>>>>>>>> operating systems is probably NFS, version 2 :-)
>>>>>>>>>>> Remember that Windows usage on a VM is not some rare use case, and
>>>>>>>>>>> it'd be a little bit of a pain from a user's perspective to have to
>>>>>>>>>>> install a third party NFS client for every VM they use. Having
>>>>>>>>>>> something supported on the VM out of the box is a better option IMO.
>>>>>>>>>> i don't think virtio-CIFS has any more support out of the box (on any
>>>>>>>>>> system) than virtio-9P.
>>>>>>>>> It doesn't, but at least network-CIFS tends to work ok and is the
>>>>>>>>> method of choice for Windows VMs - when you can setup Samba on the
>>>>>>>>> host (which as previously noted you cannot always do non-disruptively
>>>>>>>>> with current Sambas).
>>>>>>>>>
>>>>>>>>> -- Jamie
>>>>>>>>>
>>>>>>>> I think having support for both 9p and CIFS would be the best option.
>>>>>>>> In that case the user will have the option to use either one,
>>>>>>>> depending on how their guests support these filesystems. In that case
>>>>>>>> I'd prefer to work on CIFS support while the 9p effort can still go
>>>>>>>> on. I don't think both efforts are mutually exclusive.
>>>>>>>>
>>>>>>>> What do the rest of you guys think?
>>>>>>> I only noted NFS because most old OSes do not support CIFS or 9P -
>>>>>>> especially all the old unixes.
>>>>>>>
>>>>>>> I don't think old versions of MS-DOS and Windows (95, 98, ME, Nt4?)
>>>>>>> even support current CIFS.  They need extra server settings to work
>>>>>>> - such as setting passwords on the server to non-encrypted and other quirks.
>>>>>>>
>>>>>>> Meanwhile Windows Vista/2008/7 works better with SMB2, not CIFS, to
>>>>>>> properly see symlinks and hard links.
>>>>>>>
>>>>>>> So there is no really nice out of the box file service which works
>>>>>>> easily with all guest OSes.
>>>>>>>
>>>>>>> I'm guessing that out of all the filesystems, CIFS is the most widely
>>>>>>> supported in recent OSes (released in the last 10 years).  But I'm not
>>>>>>> really sure what the state of CIFS is for non-Windows, non-Linux,
>>>>>>> non-BSD guests.
>>>>>> So what? If you want to have direct host fs access, install guest drivers. If you can't, set up networking and use CIFS or NFS or whatever.
>>>>>>
>>>>>>> I'm not sure why 9P is being pursued.  Does anything much support it,
>>>>>>> or do all OSes except quite recent Linux need a custom driver for 9P?
>>>>>>> Even Linux only got the first commit in the kernel 5 years ago, so
>>>>>>> probably it was only about 3 years ago that it will have begun
>>>>>>> appearing in stable distros, if at all.  Filesystem passthrough to
>>>>>>> Linux guests installed in the last couple of years is a useful
>>>>>>> feature, and I know that for many people that is their only use of
>>>>>>> KVM, but compared with CIFS' broad support it seems like quite a
>>>>>>> narrow goal.
>>>>>> The goal is to have something simple and fast. We can fine-tune 9P to align with the Linux VFS structures, making it really little overhead (and little headache too). For Windows guests, nothing prevents us to expose yet another 9P flavor. That again would be aligned well with Windows's VFS and be slim and fast there.
>>>>>>
>>>>>> The biggest problem I see with CIFS is that it's a huge beast. There are a lot of corner cases where it just doesn't fit in. See my previous mail for more details.
>>>>>>
>>>>> As Alex mentioned, 9P is chosen for its mere simplicity and easy adaptability.
>>>>> NFS and CIFS does not give that flexibility. As we mentioned in the patch series, we are
>>>>> already seeing better numbers with 9P. Looking ahead 9P can embed KVM/QEMU knowledge
>>>>> to share physical resources like memory/cache between the host and the guest.
>>>>>
>>>>> I think looking into the windows side of 9P client would be great option too.
>>>>> The current patch on the mailing list supports .U (unix) protocol and will be introducing
>>>>> .L (Linux) variant as we move forward.
>>>> Hi Mohammed,
>>>> Please let us know once you decide on where your interest lies.
>>>> Will be glad to have you on VirtFS (9P) though. :)
>>>>
>>>>
>>>> - JV
>>>>
>>> It seems the community is more keen on getting 9P support merged than
>>> getting CIFS supported, and they have made good points to support
>>> their argument. I'm not sure whether work on this project could fit in
>>> as a GSoC project and if there is much remaining work that could make
>>> it fit in that direction. But I'd be glad to volunteer anyway :)
>> I was thinking over the wk-end what fits your schedule and your interest areas. :)
>>
>> One thing I can think of is, making NFS server export VirtFS mount on the guest to
>> the external world. This works fine now if we enable loose cache option in 9P mount.
>> But I think we should identify and make this exports work even otherwise.
>> Please let me know if this is something that you want to sign up. :)
>>
>> Thanks,
>> JV
>>
> 
> This'd be something interesting to do. I wonder if that would fit in
> the GSoC timeframe, or whether it'd be a little too short. So how long
> you'd estimate something like that would take?

I think it would take ~3PM for someone with decent VFS/NFS knowledge. 
They key is fh-to-dentry mapping. In the loose cache mode client caches 
this information .. but even in this mode we can't assume that it will be cached
forever. Need protocol amendments, client/server side changes to implement
this in the no-cache mode which can be used even in the loose cache mode when
we get a cache-miss.

Thanks,
JV
> 
> Regards,
> Mohammed
> 
> 

Re: [Qemu-devel] [GSoC 2010] Pass-through filesystem support.

[Mohammed Gamal]

On Tue, Apr 20, 2010 at 8:36 PM, jvrao <jvrao@linux.vnet.ibm.com> wrote:

... <snip> ...

>> This'd be something interesting to do. I wonder if that would fit in
>> the GSoC timeframe, or whether it'd be a little too short. So how long
>> you'd estimate something like that would take?
>
> I think it would take ~3PM for someone with decent VFS/NFS knowledge.
> They key is fh-to-dentry mapping. In the loose cache mode client caches
> this information .. but even in this mode we can't assume that it will be cached
> forever. Need protocol amendments, client/server side changes to implement
> this in the no-cache mode which can be used even in the loose cache mode when
> we get a cache-miss.
>
> Thanks,
> JV

I think I'd be glad to go for virtio-9p in GSoC. The roadmap is a
little bit hazy for me at the moment but I think we can set the goals.
I'd appreciate some pointers as to where to get more info on what to
do and if there is any relevant documentation on that matter.

Regards,
Mohammed

Re: [Qemu-devel] [GSoC 2010] Pass-through filesystem support.

[jvrao]

Mohammed Gamal wrote:
> On Tue, Apr 20, 2010 at 8:36 PM, jvrao <jvrao@linux.vnet.ibm.com> wrote:
> 
> ... <snip> ...
> 
>>> This'd be something interesting to do. I wonder if that would fit in
>>> the GSoC timeframe, or whether it'd be a little too short. So how long
>>> you'd estimate something like that would take?
>> I think it would take ~3PM for someone with decent VFS/NFS knowledge.
>> They key is fh-to-dentry mapping. In the loose cache mode client caches
>> this information .. but even in this mode we can't assume that it will be cached
>> forever. Need protocol amendments, client/server side changes to implement
>> this in the no-cache mode which can be used even in the loose cache mode when
>> we get a cache-miss.
>>
>> Thanks,
>> JV
> 
> I think I'd be glad to go for virtio-9p in GSoC. The roadmap is a
> little bit hazy for me at the moment but I think we can set the goals.
> I'd appreciate some pointers as to where to get more info on what to
> do and if there is any relevant documentation on that matter.

You can wet your feet by starting with the patch set 
"[Qemu-devel] [PATCH -V5 00/21] virtio-9p: paravirtual file system passthrough" on the mailing list.

Start QEMU on any latest distro like Fedora 12; choose mainline kernel for guest.

You can start QEMU with something like "-virtfs local,path=/tmp/,mount_tag=v_tmp" 
and mount it on the guest. ex:  mount -t 9p -o trans=virtio -o debug=0xFFFF v_tmp /mnt.

http://plan9.bell-labs.com/wiki/plan9/9p2010/index.html gives basic information on .L protocol.

You can start to play around.. by exporting NFS on top of VirtFS..

Thanks,
JV



> 
> Regards,
> Mohammed
> 
> 

Re: [Qemu-devel] [GSoC 2010] Pass-through filesystem support.

[jvrao]

Jamie Lokier wrote:
> Mohammed Gamal wrote:
>> 2- With respect to CIFS. I wonder how the shares are supposed to be
>> exposed to the guest. Should the Samba server be modified to be able
>> to use unix domain sockets instead of TCP ports and then QEMU
>> communicating on these sockets. With that approach, how should the
>> guest be able to see the exposed share? And what is the problem of
>> using Samba with TCP ports?
> 
> One problem with TCP ports is it only works when the guest's network
> is up :) You can't boot from that.  It also makes things fragile or
> difficult if the guest work you are doing involves fiddling with the
> network settings.
> 
> Doing it over virtio-serial would have many benefits.
> 
> On the other hand, Samba+TCP+CIFS does have the advantage of working
> with virtually all guest OSes, including Linux / BSDs / Windows /
> MacOSX / Solaris etc.  9P only works with Linux as far as I know.

Here is the list of 9p platforms

http://9p.cat-v.org/implementations

> 
> I big problem with Samba at the moment is it's not possible to
> instantiate multiple instances of Samba any more, and not as a
> non-root user.  That's because it contains some hard-coded paths to
> directories of run-time state, at least on Debian/Ubuntu hosts where I
> have tried and failed to use qemu's smb option, and there is no config
> file option to disable that or even change all the paths.
> 
> Patching Samba to make per-user instantiations possible again would go
> a long way to making it useful for filesystem passthrough.  Patching
> it so you can turn off all the fancy features and have it _just_ serve
> a filesystem with the most basic necessary authentication would be
> even better.
> 
> -- Jamie
> 
>