[Qemu-devel] Looking for an easy way to exchange data bidirectional between host and guest (including some suggestion)

[Qemu-devel] Looking for an easy way to exchange data bidirectional between host and guest (including some suggestion)

[Jan Marten Simons]

Ok, I talked about this issue in irc lately, but as this list will have
a larger audience I'll post this here as well. So here is the (cleaned) log:


What I want:

<jamasi> some feature I'm really missing is: a v(S)FTP sever inside the
emulated network offering a local folder of the host for R/W-access.
<pbrook> If you have sshd running on the host, you can just connect to that.
<jamasi> win-hosts usually dont have sshd running. And it would be
really appreciated, if it would work with user-net.


An idea how this could be done:

*<jamasi> so basically you could just take some tiny ftp-server source
and put it into qemu and then add a command-switch for the directory to
share. username and psw could be qemu:qemu or even left blank.*


Why the current options are no real alternative so far (either
linux-only or read-only):

<MadMerlin> can't qemu do file sharing via samba already?
<iamlost> jamasi: it already exists.
<iamlost> jamasi: virtual disks
<pbrook> iamlost: THat's only readonly.
<iamlost> pbrook: virtual floppys or virtual cdroms could be used to
transfer data from host to guest while guest is running.
<pbrook> Like I said that's still only readonly.
<iamlost> pbrook: host could read virtual hard disk from guest to
transfer data in other direction
<pbrook> Huh?
<iamlost> pbrook: so, despite being readonly, it can be made two way
<pbrook> How do you do that?
<iamlost> pbrook: in linux, via lomount.
<iamlost> as long as u mount it read-only in the host, u shouldnt have
any problems.
<pbrook> And if you're on window, or don't have root access?
<pbrook> If you're on liux you probably already have sshd running, so
can just use sftp.
<iamlost> pbrook: im sure theres something similar (like Daemon Tools)
for windows...
<jamasi> I think the vvFat-approch is pretty nice, but it blocks an
ide-channel and it somehow feels more dangerous than a simple ftp-server.
<pbrook> iamlost: daemon tools only does cdroms.
<pbrook> There are third party lomount-like things, but they tend to be
use unfriendly, and not very well supported.
<iamlost> oh
<pbrook> And you still need admin access.
<iamlost> :/
<pbrook> Plus to do that you have to shut down the guest.
<iamlost> well...u can still use virtual floppies...
<pbrook> How?
<iamlost> works fine as long as data is small enough.
<iamlost> its a little painful to use tho.
<pbrook> Very paindul, and how exactly do you mount those on a windows host?
<iamlost> when the guest writes data to a virtual floppy and ejects it,
u then write the image onto a real floppy
<iamlost> and then read that
<iamlost> i know there are tools like rawwrite.exe which can do that.
<pbrook> Most new machines don't have floppy drives.
<pbrook> Plus it's painfully slow.

*<jamasi> I think introducing some ftp-server is far less pita.
alternatively a webDAV-server would be an option.*

<iamlost> jamasi: there was a tftp sever patch some time ago
<pbrook> ftp is certainly the lowest-common-denominator simplest option.
<jamasi> this was readonly, too.
<jamasi> I'm looking for a way to exchange data with the host, without
shutting-down the guest and without needing any special conditions for
the host.
<jamasi> and for me an integrated ftp-server serving some host directory
seems like the best way to implement this.
<jamasi> sadly I'm really unskilled in c(++). Otherwise I'd try to
implement this.


So to implement the idea one would need to get the source of a tiny
gpl/bsd-ed FTP-server (does not need to have any fancy things like
usermanagement, SFTP or TFPT, ratio, quota,...), bind this to the
emulated gateway and create a new cmd-line-option to set the served
directory. A reread of the directory would be issued on a refresh-cmd
from the ftp-client on the guest.

Thanks for reading this. I'd really appreciate a patch for this
functionality, which should then be merged into cvs, imho, as this could
be an easy interface for fileexchange.

With regards,
Jan Simons

Re: [Qemu-devel] Looking for an easy way to exchange data bidirectional between host and guest (including some suggestion)

[Henrik Nordstrom]

On Fri, 3 Jun 2005, Jan Marten Simons wrote:

> Ok, I talked about this issue in irc lately, but as this list will have
> a larger audience I'll post this here as well. So here is the (cleaned) log:
>
> What I want:
>
> <jamasi> some feature I'm really missing is: a v(S)FTP sever inside the
> emulated network offering a local folder of the host for R/W-access.

Running a FTP server on the host with proper permissions is sufficient for 
this. There is plenty of FTP servers available for all platforms where 
qemu runs.

I don't quite get why everyone insists that qemu should have various 
network services built-in. There already is too much of that imho (TFTP 
and some SMB glue). The goal of qemu is to emulate the CPU and required 
hardware, not to provide full network server services.

wrt r/w access it should be pretty trivial to extend the tftp server to 
allow write access.

btw: The SMB server (Samba) provides read/write access if you insist on 
it. All you in theory need to do is to set up the password file on the 
host.

What would perhaps make sense in this discussion is to generalize the 
user-net port redirection used for the SMB access allowing it to be used 
in a more general manner, then perhaps even ripping out the smb (Samba) 
support glue and replacing it by a wrapper setting things up as suitable 
using the general slirp pipe support. So instead of having the smb support 
glue hardcoded within the qemu binary just have it start the suitable 
daemon wrapper when the port in question is accessed. Apart for allowing 
it to be used for other services than SMB it would also allow the SMB 
support to be tailored more closely to your needs.

Regards
Henrik

Re: [Qemu-devel] Looking for an easy way to exchange data bidirectional between host and guest (including some suggestion)

[Jim C. Brown]

On Sun, Jun 05, 2005 at 09:17:45PM +0200, Henrik Nordstrom wrote:
> On Fri, 3 Jun 2005, Jan Marten Simons wrote:
> 
> >Ok, I talked about this issue in irc lately, but as this list will have
> >a larger audience I'll post this here as well. So here is the (cleaned) 
> >log:
> >
> >What I want:
> >
> ><jamasi> some feature I'm really missing is: a v(S)FTP sever inside the
> >emulated network offering a local folder of the host for R/W-access.
> 
> Running a FTP server on the host with proper permissions is sufficient for 
> this. There is plenty of FTP servers available for all platforms where 
> qemu runs.
> 

Hmm...what if you don't have root/administrator access? It could still work if
you are determined enough, but thats not the sort of thing you want to force
onto a beginner.

> I don't quite get why everyone insists that qemu should have various 
> network services built-in. There already is too much of that imho (TFTP 
> and some SMB glue). The goal of qemu is to emulate the CPU and required 
> hardware, not to provide full network server services.
> 

I agree. TFTP might be useful if qemu one day supports netbooting, though.

> What would perhaps make sense in this discussion is to generalize the 
> user-net port redirection used for the SMB access allowing it to be used 
> in a more general manner, then perhaps even ripping out the smb (Samba) 
> support glue and replacing it by a wrapper setting things up as suitable 
> using the general slirp pipe support. So instead of having the smb support 
> glue hardcoded within the qemu binary just have it start the suitable 
> daemon wrapper when the port in question is accessed. Apart for allowing 
> it to be used for other services than SMB it would also allow the SMB 
> support to be tailored more closely to your needs.

I agree, thats a very good idea.

Of course, that doesn't apply if you cant use SMB, as was this person's case.
(Does the SMB support in qemu even work on Windows hosts?)

Giving the TFTP write access is probably the way to go (iirc this person would
have used TFTP in lieu of FTP, except that write access was required which made
TFTP a non-option).

> 
> Regards
> Henrik
> 

-- 
Infinite complexity begets infinite beauty.
Infinite precision begets infinite perfection.

Re: [Qemu-devel] Looking for an easy way to exchange data bidirectional between host and guest (including some suggestion)

[Henrik Nordstrom]

On Mon, 6 Jun 2005, Jim C. Brown wrote:

> Hmm...what if you don't have root/administrator access? It could still work if
> you are determined enough, but thats not the sort of thing you want to force
> onto a beginner.

Then write a suitable wrapper install package around a suitable small FTP 
daemon allowing beinners to set it up on a non-root port using user-net 
port redirection options to hide this for the guest.

Or spend some time on the generalised user-net pipe support mentioned 
earlier allowing the SMB suppor to be more easily configured/tailored.

FTP unfortunately doesn't fit in the user-net pipe framework due to it's 
(ftp) broken design (data channel mess).

HTTP could fit quite nicely in the pipe framework, but is't terribly 
useful as a general bi-directional exchange method due to client 
requirements..

SSH in theory could fit into the pipe framework, but lack of a suitable 
server-side which can run as non-root is somewhat hindering.

rlogin/rcp/rsh fits quite nicely in the pipe framework, with some small 
modifications on the server-side to allow them to run as non-root.

Thinking of it tftp should also fit very nicely in the user-net pipe 
framework, given a suitably relaxed tftp server..

> Of course, that doesn't apply if you cant use SMB, as was this person's case.
> (Does the SMB support in qemu even work on Windows hosts?)

The SMB glue is for all hosts except Windows. On Windows you have to use 
the native SMB filesharing.

In theory it may be possible to enable the SMB glue on Windows as well, 
but this requires a working Samba on Windows..

> Giving the TFTP write access is probably the way to go (iirc this person would
> have used TFTP in lieu of FTP, except that write access was required which made
> TFTP a non-option).

Extending the existing TFTP code to also provide write access is higly 
preferable to adding a user-net FTP emulation in my eyes. Shouldn't be 
more than one or at most two screens of code.

Regards
Henrik

Re: [Qemu-devel] Looking for an easy way to exchange data bidirectional between host and guest (including some suggestion)

[Jim C. Brown]

On Tue, Jun 07, 2005 at 11:45:40PM +0200, Henrik Nordstrom wrote:
> On Mon, 6 Jun 2005, Jim C. Brown wrote:
> 
> >Hmm...what if you don't have root/administrator access? It could still 
> >work if
> >you are determined enough, but thats not the sort of thing you want to 
> >force
> >onto a beginner.
> 
> Then write a suitable wrapper install package around a suitable small FTP 
> daemon allowing beinners to set it up on a non-root port using user-net 
> port redirection options to hide this for the guest.
> 

Next to impossible to do portably. I am not sure how user-net port redirection
even enters the picture - once the FTP daemon is set up, the guest sees
the FTP server on the host (admittedly on a nonstandard port, but is that
such a big deal?).

I agree that this is a better option than shoving all that FTP server code
into qemu, but I think that this is still overkill.

> Or spend some time on the generalised user-net pipe support mentioned 
> earlier allowing the SMB suppor to be more easily configured/tailored.

Yuck...

> 
> FTP unfortunately doesn't fit in the user-net pipe framework due to it's 
> (ftp) broken design (data channel mess).

Can you explain what you mean here?

<list of protocols to be used via user-net deleted>

Making any of these builtin is overkill considering what we already have.
Setting these up externally is less overkill, but still...

> 
> >Of course, that doesn't apply if you cant use SMB, as was this person's 
> >case.
> >(Does the SMB support in qemu even work on Windows hosts?)
> 
> The SMB glue is for all hosts except Windows. On Windows you have to use 
> the native SMB filesharing.
> 
> In theory it may be possible to enable the SMB glue on Windows as well, 
> but this requires a working Samba on Windows..
> 

I meant using native Windows SMB functionablity, not Samba ... but someone
else replied saying that would only work if given administrator access,
which makes it a moot point.

So this is not a portable solution. Samba may not be the best solution anyways,
since it provides a Windows-specific network filesystem...though Samba 3
provides Unix extensions.

> >Giving the TFTP write access is probably the way to go (iirc this person 
> >would
> >have used TFTP in lieu of FTP, except that write access was required which 
> >made
> >TFTP a non-option).
> 
> Extending the existing TFTP code to also provide write access is higly 
> preferable to adding a user-net FTP emulation in my eyes. Shouldn't be 
> more than one or at most two screens of code.
> 

Right, I'm looking into doing that right now.

> Regards
> Henrik
> 
> 
> _______________________________________________
> Qemu-devel mailing list
> Qemu-devel@nongnu.org
> http://lists.nongnu.org/mailman/listinfo/qemu-devel

-- 
Infinite complexity begets infinite beauty.
Infinite precision begets infinite perfection.

Re: [Qemu-devel] Looking for an easy way to exchange data bidirectional between host and guest (including some suggestion)

[Henrik Nordstrom]

On Wed, 8 Jun 2005, Jim C. Brown wrote:

> Next to impossible to do portably. I am not sure how user-net port redirection
> even enters the picture - once the FTP daemon is set up, the guest sees
> the FTP server on the host (admittedly on a nonstandard port, but is that
> such a big deal?).

With the user-net port redirection the wrapper can automate the task, 
dynamically assigning a port to the FTP server and map this on port 21 of 
the router.

>> FTP unfortunately doesn't fit in the user-net pipe framework due to it's
>> (ftp) broken design (data channel mess).
>
> Can you explain what you mean here?

The user-net pipe support when seeing a new TCP connection to the piped 
port (as used for the SMB server) forks a child process and connects the 
TCP connection via a pipe to this child process sort of as if it was 
started by inetd.

Problem is that FTP uses more than one TCP connection (the main control 
channel, plus one data channel per transfer) and this isn't supported by 
the slirp pipe support. For this kind of protocols you have to use port 
mapping or forwarding.

The real FTP server only works because it is running on a real TCP/IP, 
allowing it to open additional ports which is bidirectionally reachable 
for the client (direction depending on which PORT/PASV mode is used)

Regards
Henrik

Re: [Qemu-devel] Looking for an easy way to exchange data bidirectional between host and guest (including some suggestion)

[Jim C. Brown]

On Thu, Jun 09, 2005 at 01:56:09PM +0200, Henrik Nordstrom wrote:
> On Wed, 8 Jun 2005, Jim C. Brown wrote:
> 
> >Next to impossible to do portably. I am not sure how user-net port 
> >redirection
> >even enters the picture - once the FTP daemon is set up, the guest sees
> >the FTP server on the host (admittedly on a nonstandard port, but is that
> >such a big deal?).
> 
> With the user-net port redirection the wrapper can automate the task, 
> dynamically assigning a port to the FTP server and map this on port 21 of 
> the router.
> 

Ok this might be slightly useful. Still, any sane client should support using
a non-standard port, so this isn't a major issue.

> >>FTP unfortunately doesn't fit in the user-net pipe framework due to it's
> >>(ftp) broken design (data channel mess).
> >
> >Can you explain what you mean here?
> 
> The user-net pipe support when seeing a new TCP connection to the piped 
> port (as used for the SMB server) forks a child process and connects the 
> TCP connection via a pipe to this child process sort of as if it was 
> started by inetd.
> 
> Problem is that FTP uses more than one TCP connection (the main control 
> channel, plus one data channel per transfer) and this isn't supported by 
> the slirp pipe support. For this kind of protocols you have to use port 
> mapping or forwarding.
> 
> The real FTP server only works because it is running on a real TCP/IP, 
> allowing it to open additional ports which is bidirectionally reachable 
> for the client (direction depending on which PORT/PASV mode is used)
> 
> Regards
> Henrik
> 

I agree. While I believe this can be worked around (given a cooperative ftpd),
having a child process for every data connection is really not worth the
trouble. It's just too ugly. Besides, not every system will have an ftpd
installed.

I don't see this as a difficult problem for a builtin FTP server though,
provided that passive mode is used. If the ftp server ran on its own ip
address, then qemu could simply set up new servers on new ports for the
new data connections and have the client connect to them. Active mode
is a little harder to do right, but thats hard to get right through a NAT
connection anyways.

-- 
Infinite complexity begets infinite beauty.
Infinite precision begets infinite perfection.

Re: [Qemu-devel] Looking for an easy way to exchange data bidirectional between host and guest (including some suggestion)

[Christian MICHON]

On 6/11/05, Jim C. Brown <jma5@umd.edu> wrote:
> I agree. While I believe this can be worked around (given a cooperative ftpd),
> having a child process for every data connection is really not worth the
> trouble. It's just too ugly. Besides, not every system will have an ftpd
> installed.

since we already have a 32Mb limited tftp, could we make it first
read-write? at least, this would validate what's theoretically
possible, and we could use this rw tftp to develop/debug the
later ftp.

I had a look at the code, and can't figure out where the original
tftp command like 'get' and 'put' are parsed inside the tftp.
Apparently, it's just a question of recognising the 'put' cmd,
then open/create a file on the host matching the tftp
arguments, and that'll be it.

Magnus would be the best person to do this, since it's his
code.

Why do I still think rw tftp could be of any use ?

I personally am using on XP host linux guest doing
kernel compilation. Once the kernel is done, I have 2
scenario:

1) move the compiled kernel to a vfat partition, shutdown
the qemu guest, use winimage to extract the kernel, launch
another qemu guest to test my latest kernel

2) lauch qemu within qemu. Slow square time :( no kqemu for
windows hosts yet

The 3rd possibility would be to use a tftp put back to the host,
and launch a second qemu guest in parallel.

Christian

Re: [Qemu-devel] Looking for an easy way to exchange data bidirectional between host and guest (including some suggestion)

[Henrik Nordstrom]

On Sat, 11 Jun 2005, Christian MICHON wrote:

> I had a look at the code, and can't figure out where the original
> tftp command like 'get' and 'put' are parsed inside the tftp.
> Apparently, it's just a question of recognising the 'put' cmd,
> then open/create a file on the host matching the tftp
> arguments, and that'll be it.

The TFTP server is in slirp/tftp.c

The TFTP protocol is defined in RFC1350 / STD0033.

TFTP Write is essentially a mirror of read.

   client    server
   WRQ  ->   ACK/ERROR
   DATA ->   ACK/ERROR

Where TFPT Read looks like

   client    server
   RRQ  ->   DATA/ERROR
   ACK  ->   DATA/ERROR
   ACK (on final DATA segment)

> 1) move the compiled kernel to a vfat partition, shutdown
> the qemu guest, use winimage to extract the kernel, launch
> another qemu guest to test my latest kernel
>
> 2) lauch qemu within qemu. Slow square time :( no kqemu for
> windows hosts yet
>
> The 3rd possibility would be to use a tftp put back to the host,
> and launch a second qemu guest in parallel.

Or

4) Set up user-net port redirection allowing you to FTP into the guest 
from your XP host.

5) As 4 but using tun/tap networking instead (no need to fiddle with port 
redirection, you have access to all of the guest networking)

6) Run a FTP daemon on your host, and FTP to this from your guest.

7) Email the kernel to yourself from within the guest.

8) Use ext2 tools for to extract the kernel from the guest filesysem.

And a number of other possibilities.

Regards
Henrik

Re: [Qemu-devel] Looking for an easy way to exchange data bidirectional between host and guest (including some suggestion)

[Christian MICHON]

> 6) Run a FTP daemon on your host, and FTP to this from your guest.

I tried this one, since it looked the easiest.
I installed war-ftpd, which is working well inside the host.

I can now connect from the ftp guest (linux inside qemu), but no file
transfer is possible. Actually, even a "dir" does not work.

Yet, I can at least connect. I tested the ftpd functionality from the host,
all is ok. I can't figure out what is wrong, except somebody mentionned
recently you need 2 connections over slirp to do ftp, which is not
possible.

Now I need to look for a tftpd server for XP host...
Still investigating.

Thanks for the pointers on the tftp source. I'll look into it if the tftpd
xp host server does not work out.

Christian

Re: [Qemu-devel] Looking for an easy way to exchange data bidirectional between host and guest (including some suggestion)

[Jernej Simonèiè]

On Monday, June 13, 2005, 10:59:40, Christian MICHON wrote:

> I can now connect from the ftp guest (linux inside qemu), but no file
> transfer is possible. Actually, even a "dir" does not work.

Did you try passive mode?

-- 
< Jernej Simoncic ><><><><>< http://deepthought.ena.si/ >

The bigger they are, the harder it is to see your shoes.
       -- Dolly Parton's Principle

Re: [Qemu-devel] Looking for an easy way to exchange data bidirectional between host and guest (including some suggestion)

[Christian MICHON]

> Did you try passive mode?

nice suggestion.
passive mode off: "dir" lasts forever...
passive mode on: "dir" give "connection refused".

any idea?

Christian

Re: [Qemu-devel] Looking for an easy way to exchange data bidirectional between host and guest (including some suggestion)

[John R. Hogerhuis]

On Mon, 2005-06-13 at 14:52 +0200, Christian MICHON wrote:
> > Did you try passive mode?
> 
> nice suggestion.
> passive mode off: "dir" lasts forever...
> passive mode on: "dir" give "connection refused".
> 
> any idea?
> 
> Christian
> 

Suggest some packet sniffs to see who is getting stuck.

-- John.

Re: [Qemu-devel] Looking for an easy way to exchange data bidirectional between host and guest (including some suggestion)

[Christian MICHON]

how do i do this?
from the windows host or from the linux guest?
any special sw to install/setup?

I tried to use a tftp server on the windows host. It's functional,
but the linux guest cannot reach it. I suspect the tftp over slirp
may interfere here.

Christian

On 6/14/05, John R. Hogerhuis <jhoger@pobox.com> wrote:
> > any idea?
> >
> > Christian
> >
> 
> Suggest some packet sniffs to see who is getting stuck.
> 
> -- John.
> 
> 


-- 
Christian

Re: [Qemu-devel] Looking for an easy way to exchange data bidirectional between host and guest (including some suggestion)

[Henrik Nordstrom]

On Tue, 14 Jun 2005, Christian MICHON wrote:

> how do i do this?
> from the windows host or from the linux guest?

Both, and compare them.

> any special sw to install/setup?

tcpdump -X on Linux, or ethereal.

On Windows there is a sniffer included in Windows 2000 and later. 
Available as a download (or in the resource kit, I don't remember) for 
NT4. The name slips my mind right now (not a Windows user).

Regards
Henrik

Re: [Qemu-devel] Looking for an easy way to exchange data bidirectional between host and guest (including some suggestion)

[Jim C. Brown]

On Mon, Jun 13, 2005 at 02:52:25PM +0200, Christian MICHON wrote:
> > Did you try passive mode?
> 
> nice suggestion.
> passive mode off: "dir" lasts forever...

Because in active mode, the ftp server attempts to connect to a port on the
ftp client, in the guest. Due to the limitations of slirp, this is not possible
(in theory one could use redir support to work around this, but this is not
an easy task).

dir hangs because the client is waiting for the server to connect, which it
cant.

> passive mode on: "dir" give "connection refused".
> 
> any idea?

It is not because of slirp. That should have worked fine. Perhaps there are
firewall issues...but this is slirp, so to the host, the request looks like
its coming from itself. Strange.

> 
> Christian
> 

-- 
Infinite complexity begets infinite beauty.
Infinite precision begets infinite perfection.

Re: [Qemu-devel] Looking for an easy way to exchange data bidirectional between host and guest (including some suggestion)

[Henrik Nordstrom]

On Tue, 14 Jun 2005, Jim C. Brown wrote:

> Because in active mode, the ftp server attempts to connect to a port on the
> ftp client, in the guest. Due to the limitations of slirp, this is not possible
> (in theory one could use redir support to work around this, but this is not
> an easy task).

slirp is supposed to automatically "NAT" this on FTP connections on port 
21, but perhaps there is something fishy with the version embedded in 
qemu..

When you FTP to the virtual address then this tranlation is required both 
ways (both PORT & PASV). The difference is mainly when you FTP to the real 
host address.

Regards
Henrik

Re: [Qemu-devel] Looking for an easy way to exchange data bidirectional between host and guest (including some suggestion)

[Henrik Nordstrom]

On Sat, 11 Jun 2005, Jim C. Brown wrote:

>> With the user-net port redirection the wrapper can automate the task,
>> dynamically assigning a port to the FTP server and map this on port 21 of
>> the router.
>
> Ok this might be slightly useful. Still, any sane client should support using
> a non-standard port, so this isn't a major issue.

It's mainly a usability issue.

By using the SLIRP port-redirection the port assignment and FTP protocol 
support can be automated, and all the user is to remember is that he 
should FTP to the router IP.

> I agree. While I believe this can be worked around (given a cooperative ftpd),
> having a child process for every data connection is really not worth the
> trouble. It's just too ugly. Besides, not every system will have an ftpd
> installed.

The main issue to over come is how you work with slirp. The "external 
helper application" mode of slirp is simply not designed for protocols 
requiring more than one connection.

Getting this to work under the "pipe" interface of SLIRP requires far more 
than a cooperative FTP daemon, it also requires a new protocol to be 
defined between SLIRP and this cooperative FTP daemon..  and in such case 
it is much better to simply stick to the already well-established FTP 
protocol imho (which calls for port redirection as discussed before).

> I don't see this as a difficult problem for a builtin FTP server though,
> provided that passive mode is used.

For a built-in FTP server passive or active mode should not matter. You 
will then be running below the SLIRP TCP socket layer, not standard TCP/IP 
sockets.

> If the ftp server ran on its own ip
> address, then qemu could simply set up new servers on new ports for the
> new data connections and have the client connect to them.

You already have the router virtual IP address under your (or SLIRP) full 
control. No need to invent additional addresses imho.

> Active mode is a little harder to do right, but thats hard to get right 
> through a NAT connection anyways.

I don't see why active should be much harder than passive. SLIRP already 
have support for both accepting and making connections both ways. Just 
use what there is. There will be a somewhat hefty state machinery involved 
however.. writing an FTP daemon within SLIRP is considerably more complex 
than the same thing as a stand-alone daemon.

On another note reshapign the already existing (but disabled) rsh 
emulation to fake a "trusted" login to the host should be pretty trivial, 
allowing rcp etc to work fine on UNIX hosts. Some small modifications of 
the existing code is required to grab both usernames (for .rhosts 
processing), some .rhosts processing for security and also some slight 
modifications in rsh_exec to fork a shell rather than trying to rsh 
somewhere else..

Regards
Henrik

Re: [Qemu-devel] Looking for an easy way to exchange data bidirectional between host and guest (including some suggestion)

[Paul Brook]

On Monday 13 June 2005 01:02, Henrik Nordstrom wrote:

> > If the ftp server ran on its own ip
> > address, then qemu could simply set up new servers on new ports for the
> > new data connections and have the client connect to them.
>
> You already have the router virtual IP address under your (or SLIRP) full
> control. No need to invent additional addresses imho.

I don't think this is a good idea. The current behaviour is to automatically 
forward connections to the router on to the host. INMO this is very useful 
and desirable behaviour, so any new services should be added on a different 
virtual IP. We already use additional addresses for dns, tftp and SMB.

> I don't see why active should be much harder than passive. SLIRP already 
> have support for both accepting and making connections both ways. Just 
> use what there is. There will be a somewhat hefty state machinery involved 
> however.. writing an FTP daemon within SLIRP is considerably more complex 
> than the same thing as a stand-alone daemon.

It doesn't have to be. You can use the same trick as the fork_exec code does. 
ie. the ftp code listens on a localhost socket, and use the normal slirp 
connection forwarding.

Paul

Re: [Qemu-devel] Looking for an easy way to exchange data bidirectional between host and guest (including some suggestion)

[marten]

> On Fri, 3 Jun 2005, Jan Marten Simons wrote:
>
>> Ok, I talked about this issue in irc lately, but as this list will have
>> a larger audience I'll post this here as well. So here is the (cleaned)
>> log:
>>
>> What I want:
>>
>> <jamasi> some feature I'm really missing is: a v(S)FTP sever inside the
>> emulated network offering a local folder of the host for R/W-access.
>
> Running a FTP server on the host with proper permissions is sufficient for
> this. There is plenty of FTP servers available for all platforms where
> qemu runs.

As Jim Brown said it's not that easy to use a FTP-server on the host, if
you've got no root/admin-access.

Think of this scenario: A DVD with qemu and some disc-image and some
start-up-script for linux and an autostart.exe for win-host which will
fire-up the disk-image really easy. That way you can have a sand-boxed
standard environment which can be used on either windows or linux hosts
without the need to configure those. Now you should preconfigure your
guest to save personal information somewhere. So I'd like to have an
"exchange"-director inside the $home of the current user which should be
avialiable inside the virtual-pc for read/write. I think it would be the
easiest way to offer it via an integrated FTP-server. The advantages are:
- no ide-channel is blocked (unlike vvFat)
- ftp IS a very common protocol (is there SMB for exotic OS?)
- as the server is only virtual, you don't have to care about security-issues
- the server only runs, when it is needed
- implementaion should be easy (get some open-source FTP, strip it down
and change its binding to an interface like it was done with the internal
dhcp)
- the data-transfer resides inside the guest, which would be more
intuitive to the non-technical user when working in the VM.

> btw: The SMB server (Samba) provides read/write access if you insist on
> it. All you in theory need to do is to set up the password file on the
> host.

The Samba-wrapper does not work on Win-on-Win situations

> What would perhaps make sense in this discussion is to generalize the
> user-net port redirection used for the SMB access allowing it to be used
> in a more general manner, then perhaps even ripping out the smb (Samba)
> support glue and replacing it by a wrapper setting things up as suitable
> using the general slirp pipe support. So instead of having the smb support
> glue hardcoded within the qemu binary just have it start the suitable
> daemon wrapper when the port in question is accessed. Apart for allowing
> it to be used for other services than SMB it would also allow the SMB
> support to be tailored more closely to your needs.

This would be desirable, too. But I think that this would be far more
difficult to implement.

Have a nice day,
Jan

Re: [Qemu-devel] Looking for an easy way to exchange data bidirectional between host and guest (including some suggestion)

[Jim C. Brown]

On Mon, Jun 06, 2005 at 08:11:08PM +0200, marten@xtal.rwth-aachen.de wrote:
> The advantages are:
> - no ide-channel is blocked (unlike vvFat)

I don't believe this is a major issue. Alias, vvFat is readonly and anyways not
suitable for the purposes of every OS (e.g. copy from Linux guest to Linux
host, preserving ownership).

> - ftp IS a very common protocol (is there SMB for exotic OS?)
> - as the server is only virtual, you don't have to care about security-issues
> - the server only runs, when it is needed
> - implementaion should be easy (get some open-source FTP, strip it down
> and change its binding to an interface like it was done with the internal
> dhcp)
> - the data-transfer resides inside the guest, which would be more
> intuitive to the non-technical user when working in the VM.
> 

Like I said, modifying TFTP for R/W would be a good option. It's already there,
the "miminalists" can't complain about having it removed (e.g. it may one day
be used to support "virtual" netboots), and one can use ftp clients for the
tftp server (I think).

> > btw: The SMB server (Samba) provides read/write access if you insist on
> > it. All you in theory need to do is to set up the password file on the
> > host.
> 
> The Samba-wrapper does not work on Win-on-Win situations
> 

But windows host has native SMB support. Surely the wrapper could be altered
to support using SMB via win32 ?

> 
> Have a nice day,
> Jan
> 
> 
> 
> _______________________________________________
> Qemu-devel mailing list
> Qemu-devel@nongnu.org
> http://lists.nongnu.org/mailman/listinfo/qemu-devel
> 

-- 
Infinite complexity begets infinite beauty.
Infinite precision begets infinite perfection.

Re: [Qemu-devel] Looking for an easy way to exchange data bidirectional between host and guest (including some suggestion)

[Jan Marten Simons]

>> - no ide-channel is blocked (unlike vvFat)
>
> I don't believe this is a major issue. Alias, vvFat is readonly and
> anyways not
> suitable for the purposes of every OS (e.g. copy from Linux guest to Linux
> host, preserving ownership).

Well for the current project I'm working on, I'm running short of
ide-channels in some situations:
I've one ro-diskimg containig the OS then there is an exchangeable
ro-discimg containig programms for different purposes, then there is the
rw-diskimg where the user can store his information. And sometimes a
software needs to have a cd in the drive as well.

> Like I said, modifying TFTP for R/W would be a good option. It's already
there,
> the "miminalists" can't complain about having it removed (e.g. it may
one day
> be used to support "virtual" netboots), and one can use ftp clients for
> the tftp server (I think).

Well the TFTP-server should get r/w, too, but if I get it right you'd have
to access the data on the guest from the host, which might lead to
situation where users shutdown the guest and then try to get their
(then lost) data from it via TFTP, which has to be installed on the host.

>> The Samba-wrapper does not work on Win-on-Win situations
>>
>
> But windows host has native SMB support. Surely the wrapper could be
> altered to support using SMB via win32 ?

I guess it could be modified, but then you'd have to share some
directories  via smb either on the host or the guest, which would be some
additional requierement, which would destroy an out-of-the-box use (insert
DVD in any workstation on the lan, get your working sandboxed virtual
desktop)

Have a nice day,
  Jan

Re: [Qemu-devel] Looking for an easy way to exchange data bidirectional between host and guest (including some suggestion)

[Johannes Schindelin]

Hi,

On Tue, 7 Jun 2005, Jan Marten Simons wrote:

> >> - no ide-channel is blocked (unlike vvFat)
> >
> > I don't believe this is a major issue. Alias, vvFat is readonly and
> > anyways not
> > suitable for the purposes of every OS (e.g. copy from Linux guest to Linux
> > host, preserving ownership).
>
> Well for the current project I'm working on, I'm running short of
> ide-channels in some situations:
>
> [...]

That is why I tried to support floppy images. The basic difference is:
floppies only have one partition. AFAIK there is no theoretical limit to
1.44M floppies. The support is not yet complete, though...

Hth, Dscho

Re: [Qemu-devel] Looking for an easy way to exchange data bidirectional between host and guest (including some suggestion)

[Henrik Nordstrom]

On Tue, 7 Jun 2005, Jan Marten Simons wrote:

> Well the TFTP-server should get r/w, too, but if I get it right you'd have
> to access the data on the guest from the host, which might lead to
> situation where users shutdown the guest and then try to get their
> (then lost) data from it via TFTP, which has to be installed on the host.

The TFTP support in qemu provides the guest TFTP access to a directory on 
the host.

If the guest is not running then you have to resort to extract the data 
you are looking for from the guest drive image(s). Not really qemu 
dependent.

Regards
Henrik

Re: [Qemu-devel] Looking for an easy way to exchange data bidirectional between host and guest (including some suggestion)

[Jim C. Brown]

On Tue, Jun 07, 2005 at 01:38:14AM +0200, Jan Marten Simons wrote:
> > Like I said, modifying TFTP for R/W would be a good option. It's already
> there,
> > the "miminalists" can't complain about having it removed (e.g. it may
> one day
> > be used to support "virtual" netboots), and one can use ftp clients for
> > the tftp server (I think).
> 
> Well the TFTP-server should get r/w, too, but if I get it right you'd have
> to access the data on the guest from the host, which might lead to
> situation where users shutdown the guest and then try to get their
> (then lost) data from it via TFTP, which has to be installed on the host.

I'm not sure I understand what you mean here.

TFTP server support is builtin to qemu, once guest goes down so does TFTP.
To access host data from guest is then no problem, but to access data on guest
from host, you need to explicitly upload that data on the TFTP server - which
will copy the data on the host.

-- 
Infinite complexity begets infinite beauty.
Infinite precision begets infinite perfection.

Re: [Qemu-devel] Looking for an easy way to exchange data bidirectional between host and guest (including some suggestion)

[Henrik Nordstrom]

On Mon, 6 Jun 2005, Jim C. Brown wrote:

> Like I said, modifying TFTP for R/W would be a good option. It's already there,
> the "miminalists" can't complain about having it removed (e.g. it may one day
> be used to support "virtual" netboots), and one can use ftp clients for the
> tftp server (I think).

tftp requires a tftp client. It shares nothing in common with FTP.

> But windows host has native SMB support. Surely the wrapper could be altered
> to support using SMB via win32 ?

The wrapper is just a thin user-net pipe glue to automatically start Samba 
when the SMB port is accessed. In the Windows world the equivalence is a 
port redirection of the SMB port to the SMB service on the host, or 
talking to the host address directly.. and requires you to have privileges 
to set up the share permissions on the Windows host.

Regards
Henrik

Re: [Qemu-devel] Looking for an easy way to exchange data bidirectional between host and guest (including some suggestion)

[John R. Hogerhuis]

I can think of some reasons for a non-native file service that is
instead built into QEMU:

a) One interface to all clients outside of QEMU. Lowers the learning
curve for dealing with different enviroments; at least for some raw file
access they are all accessible in the same way. You can lower
configuration close to zero-config, just as was done with user mode
networking. This allows to control the complexity of the configuration,
as opposed to Samba, Windows, NFS, issues in each particular
circumstance. Don't underestimate the value of expedience.

c) Vintage OSes where being networked is atypical or for which network
compatibility with QEMU is currently problematic like old versions of
DOS

d) Non-desktop emulation of non-networked embedded devices.

e) Scripts to be used across farms of QEMU virtual machines will have
more commonalities even across different OSes. The event and content of
a file popping up on a machine in a given directory is a very basic form
of inter-process communication common in business operations. Having one
way that always or almost always works with QEMU virtual machines will
be good for QEMU strategically. Look at VmWare... purchased recently by
EMC corporation. Their product line is currently geared toward packing
multiple 'virtual servers" onto beefy real servers.

-- John.

Re: [Qemu-devel] Looking for an easy way to exchange data bidirectional between host and guest (including some suggestion)

[Christian MICHON]

all these discussions around tftp looked nice and sweet. So I told
myself, instead of using winimage (shareware), why not transferring
files "live".

Yet, when you try on a windows host to exchange files with a linux
guest (inside qemu and with -tftp option), all you get is:

Error code 2: Access violation

is -tftp today working only on linux hosts ? I also read somewhere
the transfer is limited to files lesser than 64k. Can anyone confirm?

Christian

Re: [Qemu-devel] Looking for an easy way to exchange data bidirectional between host and guest (including some suggestion)

[Henrik Nordstrom]

On Tue, 7 Jun 2005, Christian MICHON wrote:

> all these discussions around tftp looked nice and sweet. So I told
> myself, instead of using winimage (shareware), why not transferring
> files "live".
>
> Yet, when you try on a windows host to exchange files with a linux
> guest (inside qemu and with -tftp option), all you get is:
>
> Error code 2: Access violation

Did you specify the correct path? The full path on the host has to be 
specified, exacly as given to qemu. In addition only "UNIX" style paths is 
supported.

> is -tftp today working only on linux hosts?

>From what I can tell it should work on all hosts, but there is some 
security issues on Windows and the path name support is not that good for 
Windows either..

> I also read somewhere
> the transfer is limited to files lesser than 64k. Can anyone confirm?

Not quite, but TFTP by protocol design limits filesize to 2^16-1 blocks of 
512 bytes or 32 MB minus 512 bytes (33553920 bytes).

Regards
Henrik

Re: [Qemu-devel] Looking for an easy way to exchange data bidirectional between host and guest (including some suggestion)

[Christian MICHON]

I used 3 ways, either valid with msys or dos shell.
all fail :(
I can ping 10.0.2.2 perfectly, I even tried as administrator inside a dos
prompt.

the file I'm trying to get inside the guest is barely 58 bytes long

Anyone ever got tftp to work inside a XP host ?

Christian

> > Error code 2: Access violation
> 
> Did you specify the correct path? The full path on the host has to be
> specified, exacly as given to qemu. In addition only "UNIX" style paths is
> supported.
> 
> > is -tftp today working only on linux hosts?
> 
> From what I can tell it should work on all hosts, but there is some
> security issues on Windows and the path name support is not that good for
> Windows either..

Re: [Qemu-devel] Looking for an easy way to exchange data bidirectional between host and guest (including some suggestion)

[Henrik Nordstrom]

On Wed, 8 Jun 2005, Christian MICHON wrote:

> I used 3 ways, either valid with msys or dos shell.
> all fail :(
> I can ping 10.0.2.2 perfectly, I even tried as administrator inside a dos
> prompt.

Good.

> the file I'm trying to get inside the guest is barely 58 bytes long
>
> Anyone ever got tftp to work inside a XP host ?

I don't have any to test on, but the code is extremely simple and not OS 
dependeng in any manner except the small detail that UNIX style paths must 
be used.

Where is the file on the host?

What tftp argument did you gave to qemu when you started it?

What path did you try to retreive from the guest?

Regards
Henrik

Re: [Qemu-devel] Looking for an easy way to exchange data bidirectional between host and guest (including some suggestion)

[Christian MICHON]

I made some progress based on your suggestions :)

> I don't have any to test on, but the code is extremely simple and not OS
> dependeng in any manner except the small detail that UNIX style paths must
> be used.
> 
> Where is the file on the host?

c:\q\w

> 
> What tftp argument did you gave to qemu when you started it?

-tftp "/q"
inside a msdos prompt => it's correct according to docs

> 
> What path did you try to retreive from the guest?

this is the location of my mistake: I did "get w" instead of "get /q/w"

now it works in "get" mode only. Is there anywhere a patch to get rw
access to tftp, as well as upping the limit of 32M ?

I indeed got a "transfer timeout" after 33553920 bytes :(

Christian

Re: [Qemu-devel] Looking for an easy way to exchange data bidirectional between host and guest (including some suggestion)

[Henrik Nordstrom]

On Wed, 8 Jun 2005, Christian MICHON wrote:

> this is the location of my mistake: I did "get w" instead of "get /q/w"
>
> now it works in "get" mode only.

Good.

> Is there anywhere a patch to get rw
> access to tftp

As already said the TFTP server is currently read-only, however 
implementing write access is not a difficult task, probably about a 
screenful of code most of which is a copy of the read counterparts.

>  as well as upping the limit of 32M ?

No, this is a protocol limitation of TFTP.

But you can split the file in several chunks, each no larger than 32MB - 
512bytes.

Regards
Henrik

Re: [Qemu-devel] Looking for an easy way to exchange data bidirectional between host and guest (including some suggestion)

[Jan Marten Simons]

> Not quite, but TFTP by protocol design limits filesize to 2^16-1 blocks
> of 512 bytes or 32 MB minus 512 bytes (33553920 bytes).

Well, if this is the case FTP should realy be added as an alternative
protocol.

And the more I think about vvFAT, the more I think it's too complex to get
this working bidirectional and secure in regard to file integrity.

Jan

Re: [Qemu-devel] Looking for an easy way to exchange data bidirectional between host and guest (including some suggestion)

[Jim C. Brown]

On Thu, Jun 09, 2005 at 12:31:18AM +0200, Jan Marten Simons wrote:
> > Not quite, but TFTP by protocol design limits filesize to 2^16-1 blocks
> > of 512 bytes or 32 MB minus 512 bytes (33553920 bytes).
> 
> Well, if this is the case FTP should realy be added as an alternative
> protocol.
> 
> And the more I think about vvFAT, the more I think it's too complex to get
> this working bidirectional and secure in regard to file integrity.
> 
> Jan
> 

I've been convinced. I'll take a look at it and see what can be done for a
builtin ftp server.

-- 
Infinite complexity begets infinite beauty.
Infinite precision begets infinite perfection.

Re: [Qemu-devel] Looking for an easy way to exchange data bidirectional between host and guest (including some suggestion)

[Jan Marten Simons]

Jim C. Brown wrote:

>I've been convinced. I'll take a look at it and see what can be done for a
>builtin ftp server.
>
Hi Jim,


Are there any results, yet?
I'd like to help with testing this feature, as I cannot really help with
coding.

Jan

Re: [Qemu-devel] Looking for an easy way to exchange data bidirectional between host and guest (including some suggestion)

[Jim C. Brown]

On Tue, Jun 21, 2005 at 04:17:08PM +0200, Jan Marten Simons wrote:
> Jim C. Brown wrote:
> 
> >I've been convinced. I'll take a look at it and see what can be done for a
> >builtin ftp server.
> >
> Hi Jim,
> 
> 
> Are there any results, yet?
> I'd like to help with testing this feature, as I cannot really help with
> coding.
> 
> Jan
> 

Not a lot yet. I have a minimal ftp server that would be perfect to use with
qemu, with much of the code rewritten. However, to make it interface with slirp
a lot more work is needed (basicly the ftp server uses sockets while slirp works
with tcp/ip packets directly. Thus I have to write a big interface layer (a sort
of BSD sockets library on top of the slirp tcp/ip stack).

The other approach that I've opted out of is to hack slirp to have support for
the ftp protocol and run the server externally thru real sockets. If you or
anyone else wants to try this, I'm willing to help a little.

-- 
Infinite complexity begets infinite beauty.
Infinite precision begets infinite perfection.

Re: [Qemu-devel] Looking for an easy way to exchange data bidirectional between host and guest (including some suggestion)

[Paul Brook]

> Not a lot yet. I have a minimal ftp server that would be perfect to use
> with qemu, with much of the code rewritten. However, to make it interface
> with slirp a lot more work is needed (basicly the ftp server uses sockets
> while slirp works with tcp/ip packets directly. Thus I have to write a big
> interface layer (a sort of BSD sockets library on top of the slirp tcp/ip
> stack).

Can't you just make the ftp code listen on a loopback socket, then use the 
normal slirp forwarding code?

Paul

Re: [Qemu-devel] Looking for an easy way to exchange data bidirectional between host and guest (including some suggestion)

[Jim C. Brown]

On Tue, Jun 21, 2005 at 10:39:14PM +0100, Paul Brook wrote:
> 
> > Not a lot yet. I have a minimal ftp server that would be perfect to use
> > with qemu, with much of the code rewritten. However, to make it interface
> > with slirp a lot more work is needed (basicly the ftp server uses sockets
> > while slirp works with tcp/ip packets directly. Thus I have to write a big
> > interface layer (a sort of BSD sockets library on top of the slirp tcp/ip
> > stack).
> 
> Can't you just make the ftp code listen on a loopback socket, then use the 
> normal slirp forwarding code?
> 
> Paul
> 

Yes, as I said in my original email:

> The other approach that I've opted out of is to hack slirp to have support for
> the ftp protocol and run the server externally thru real sockets. If you or
> anyone else wants to try this, I'm willing to help a little.

There is nothing wrong with using the slirp forwarding code (it would probably
be a lot easier to get to work), I just think that its overkill to have to
open a socket (rather, one for each data connection) on the host just so qemu
can talk to itself.

-- 
Infinite complexity begets infinite beauty.
Infinite precision begets infinite perfection.

Re: [Qemu-devel] Looking for an easy way to exchange data bidirectional between host and guest (including some suggestion)

[Johannes Schindelin]

Hi,

On Thu, 9 Jun 2005, Jan Marten Simons wrote:

> > Not quite, but TFTP by protocol design limits filesize to 2^16-1 blocks
> > of 512 bytes or 32 MB minus 512 bytes (33553920 bytes).
>
> Well, if this is the case FTP should realy be added as an alternative
> protocol.

Yes, please! And just for me, could you add a microwave oven, and maybe a
Ferrari to QEmu also?

Ciao,
Dscho

Re: [Qemu-devel] Looking for an easy way to exchange data bidirectional between host and guest (including some suggestion)

[Jan Marten Simons]

Johannes Schindelin wrote:

>Yes, please! And just for me, could you add a microwave oven, and maybe a
>Ferrari to QEmu also?
>
>Ciao,
>Dscho
>  
>

Ok, actually I wouldn't mind if qemu could be used for showing-off or
getting something heated, but to get things straight:

A very simple internal FTP-server would be about 5 to 10kb of code added
in an external ftp.c. It would be a real improvement for the stated
reasons and it's no bloat, as I think quite a few people will find this
useful. So I'd like to thank Jim in advance for trying this one.

Have a nice day,
Jan

Re: [Qemu-devel] Looking for an easy way to exchange data bidirectional between host and guest (including some suggestion)

[Henrik Nordstrom]

On Mon, 6 Jun 2005, John R. Hogerhuis wrote:

> I can think of some reasons for a non-native file service that is
> instead built into QEMU:

vvfat?

> e) Scripts to be used across farms of QEMU virtual machines will have
> more commonalities even across different OSes. The event and content of
> a file popping up on a machine in a given directory is a very basic form
> of inter-process communication common in business operations.

Using something like vvfat for bi-directional IPC would be a very bad 
move. Most guests doesn't guarantee data written on a local drive gets 
commited to the drive in a timely fashion (can be anywhere from seconds to 
many minutes before the data is actually written).

If you need bi-directional communication with a running guest you should 
be using a networked filesystem. Most if not all other approaches will not 
play nicely with the guest.

Even if vvfat can provide what to the guest looks like FAT block 
device access to a directory of files on the host the directory must not 
be touched in any manner while qemu is accessing it. Providing this kind 
of emulation is complex enough as it is.

What vvfat does is that it emulates a FAT formatted block device from a 
directory of files on the host. Currently this emulation is read-only 
(mostly) but it is moving towards read-write access which will when 
completed allow the guest to not only read the files but also write new 
content. But it is important to understand that write access is far from 
trivial. The vvfat driver have to piece together the block level writes 
performed by the guest and reconstruct the filestructure to know what 
files to map this to on the host. It is not like the guest tells vvfat 
which file it is writing to.

Regards
Henrik

Re: [Qemu-devel] Looking for an easy way to exchange data bidirectional between host and guest (including some suggestion)

[John R. Hogerhuis]

On Wed, 2005-06-08 at 00:15 +0200, Henrik Nordstrom wrote:
> On Mon, 6 Jun 2005, John R. Hogerhuis wrote:
> 
> > I can think of some reasons for a non-native file service that is
> > instead built into QEMU:
> 
> vvfat?
> 
> > e) Scripts to be used across farms of QEMU virtual machines will have
> > more commonalities even across different OSes. The event and content of
> > a file popping up on a machine in a given directory is a very basic form
> > of inter-process communication common in business operations.
> 
> Using something like vvfat for bi-directional IPC would be a very bad 
> move. Most guests doesn't guarantee data written on a local drive gets 
> commited to the drive in a timely fashion (can be anywhere from seconds to 
> many minutes before the data is actually written).
> 

For the kind of thing I'm referring to is not usually very time
critical. The resolution is on the order of 15 to 30 minutes to an
hour... some stores get polled for data, the accounting system imports
the data and generates some reports which feed into other systems, etc.
Nothing particularly time critical.

Often you have a process which comes through periodically polling for
files. And it's usually not bidirectional communication but I don't know
if that matters much. You are alluding to the problem with these
approaches of two different processes acting like they own the virtual
hardware. The only thing that can make it work is that in reality QEMU
actually owns the virtual drive, and just needs to present a clear
illusion to the guest that it is so. If some rules are followed by the
guest it is possible without any difficulty to write without stepping on
the guest's toes. For example if an entire drive or partition were made
read-only to the guest that would solve the problem. Without that
assumption it gets more complex depending on the file system type and
how the os treats it.

But certainly I agree it is an inherently difficult problem.

> Even if vvfat can provide what to the guest looks like FAT block 
> device access to a directory of files on the host the directory must not 
> be touched in any manner while qemu is accessing it. Providing this kind 
> of emulation is complex enough as it is.
> 
> What vvfat does is that it emulates a FAT formatted block device from a 
> directory of files on the host. Currently this emulation is read-only 
> (mostly) but it is moving towards read-write access which will when 
> completed allow the guest to not only read the files but also write new 
> content. But it is important to understand that write access is far from 
> trivial. The vvfat driver have to piece together the block level writes 
> performed by the guest and reconstruct the filestructure to know what 
> files to map this to on the host. It is not like the guest tells vvfat 
> which file it is writing to.

Of course, if I understand what you are saying it doesn't have to tell
vvfat though... all events pass through QEMU so you have a place to hook
events and there's no polling involved. The more stupid the guest's file
structure, the more feasible such a solution becomes. "Open/create" and
"write" and "seek" should be possible to recover, efficiently even, if
you maintain a hash table or other data structure which lets you go from
block # to file (or block # to free space).

I guess the event you would be missing is "close," but a idle timeout is
probably a decent heuristic. Also if the file is locked, that would have
to be mimiced.

-- John.

Re: [Qemu-devel] Looking for an easy way to exchange data bidirectional between host and guest (including some suggestion)

[Henrik Nordstrom]

On Tue, 7 Jun 2005, John R. Hogerhuis wrote:

> For the kind of thing I'm referring to is not usually very time
> critical. The resolution is on the order of 15 to 30 minutes to an
> hour... some stores get polled for data, the accounting system imports
> the data and generates some reports which feed into other systems, etc.
> Nothing particularly time critical.

Ok, so reading data from the guest may work in this situation, but you 
need to be careful to avoid races.

Updates in the other direction is harder however, unless vvfat is changed 
to emulate a floppy with floppy change notification and the guest has 
support for changing floppy "at random".

> Of course, if I understand what you are saying it doesn't have to tell
> vvfat though... all events pass through QEMU so you have a place to hook
> events and there's no polling involved. The more stupid the guest's file
> structure, the more feasible such a solution becomes. "Open/create" and
> "write" and "seek" should be possible to recover, efficiently even, if
> you maintain a hash table or other data structure which lets you go from
> block # to file (or block # to free space).
>
> I guess the event you would be missing is "close," but a idle timeout is
> probably a decent heuristic. Also if the file is locked, that would have
> to be mimiced.

The concept of "file" does not exist at this level, only blocks of data on 
a block device (hdd, floppy, cdrom etc, a long series of blocks to the 
size of the device).

To make it look like files qemu has to make it look like a block device 
with a block based filesystem on, and for write support the emulation must 
support any kind of modifications to the block level filesystem the guest 
may do.

There is several big issues involved in providing write support

a) On guest writes, the emulation must have a very good understanding of 
how the guest writes to the emulated filesystem to allow it to piece 
together the block level writes and map this to the files it is supposed 
to represent on the host.

b) On host writes it must detect changes in the directory and reflect this 
into the block level filesystem structure. You also have to battle with 
filesystem and block level caches in the guest to make the guest see the 
updated block level filesystem structure.

The only sane way of having a concept of files is to rely on the networked 
filesystems (SMB/NFS/FTP/whatever), but this obviously requires 
networking. Networked filesystem works on a file level.

Regards
Henrik

Re: [Qemu-devel] Looking for an easy way to exchange data bidirectional between host and guest (including some suggestion)

[Johannes Schindelin]

Hi,

On Wed, 8 Jun 2005, Henrik Nordstrom wrote:

> On Tue, 7 Jun 2005, John R. Hogerhuis wrote:
>
> Updates in the other direction is harder however, unless vvfat is changed
> to emulate a floppy with floppy change notification and the guest has
> support for changing floppy "at random".

The support infrastructure is already there. But there is not yet any way
to set it (ideally it should auto-detect if the emulated device is a
floppy or a hard disk). It is also not tested.

Every half-sane PC OS has support for changing the floppy "at random", as
every PC floppy device has a button which the user can (and will) push at
the worst moment.

> To make it look like files qemu has to make it look like a block device
> with a block based filesystem on, and for write support the emulation must
> support any kind of modifications to the block level filesystem the guest
> may do.
>
> There is several big issues involved in providing write support
>
> a) On guest writes, the emulation must have a very good understanding of
> how the guest writes to the emulated filesystem to allow it to piece
> together the block level writes and map this to the files it is supposed
> to represent on the host.

No. It MUST NOT have a ver good understanding of the guest writes, because
different OSes have different write strategies. The block driver MUST be
agnostic to this. That is what makes write-support so hard.

> b) On host writes it must detect changes in the directory and reflect this
> into the block level filesystem structure. You also have to battle with
> filesystem and block level caches in the guest to make the guest see the
> updated block level filesystem structure.

That is the main problem. Basically, in FAT a file gets written in three
stages: the directory entry is modified, the block list (in the File
Allocation Table) is adapted, and the actual data is written into the
blocks. Now, a seemingly good heuristic would find in which order these
entries are written, right? Wrong: The caching more often than not makes
this infeasible. While the OS really writes the three stages in the same
order, it does so only to the cache. And which cached blocks get written
out first is quite deliberate. Linux uses (very limited) caching on
floppies, too.

Ciao,
Dscho

Re: [Qemu-devel] Looking for an easy way to exchange data bidirectional between host and guest (including some suggestion)

[Henrik Nordstrom]

On Wed, 8 Jun 2005, Johannes Schindelin wrote:

>> There is several big issues involved in providing write support
>>
>> a) On guest writes, the emulation must have a very good understanding of
>> how the guest writes to the emulated filesystem to allow it to piece
>> together the block level writes and map this to the files it is supposed
>> to represent on the host.
>
> No. It MUST NOT have a ver good understanding of the guest writes, because
> different OSes have different write strategies. The block driver MUST be
> agnostic to this. That is what makes write-support so hard.

Exacly, which to me is the same thing ;-)

(having a very good understanding automatically gives the conclusion that 
the data is mainly written in block level order by the elevator, not 
filesystem structure order)

>> b) On host writes it must detect changes in the directory and reflect this
>> into the block level filesystem structure. You also have to battle with
>> filesystem and block level caches in the guest to make the guest see the
>> updated block level filesystem structure.
>
> That is the main problem. Basically, in FAT a file gets written in three
> stages: the directory entry is modified, the block list (in the File
> Allocation Table) is adapted, and the actual data is written into the
> blocks. Now, a seemingly good heuristic would find in which order these
> entries are written, right? Wrong: The caching more often than not makes
> this infeasible. While the OS really writes the three stages in the same
> order, it does so only to the cache. And which cached blocks get written
> out first is quite deliberate. Linux uses (very limited) caching on
> floppies, too.

Eh, I am talking about writes from the host side here. But what you write 
is a very accurate description of problem 'a' (understanding of how 
the guest writes to the FS and piece the writes together to allow mapping 
them back the the host files).

What I tried to refer to here is that vvfat if supporting writes from the 
host side it must monitor the mapped directory and on each host side 
initiated change of any kind (write to file, addition/removal of file, new 
subdirectory, timestamp change etc) simulate a "change floppy" event, and 
rebuild it's emulated fat structure from the host directory.

Regards
Henrik

Re: [Qemu-devel] Looking for an easy way to exchange data bidirectional between host and guest (including some suggestion)

[Johannes Schindelin]

Hi,

On Thu, 9 Jun 2005, Henrik Nordstrom wrote:

> What I tried to refer to here is that vvfat if supporting writes from the
> host side it must monitor the mapped directory and on each host side
> initiated change of any kind (write to file, addition/removal of file, new
> subdirectory, timestamp change etc) simulate a "change floppy" event, and
> rebuild it's emulated fat structure from the host directory.

Now I know what you mean. For me, it was always about the "how do I get my
guest files out" direction.

Ciao,
Dscho

Re: [Qemu-devel] Looking for an easy way to exchange data bidirectional between host and guest (including some suggestion)

[Jim C. Brown]

On Mon, Jun 06, 2005 at 04:54:25PM -0700, John R. Hogerhuis wrote:
> I can think of some reasons for a non-native file service that is
> instead built into QEMU:
> 
<reasons snipped>
> 
> -- John.
> 

I agree with you there, especially for non-networked OS support. A qemu-specific
guest program will take in a file and tell qemu what the filename should be and
give it the size/contents of said file. Another such program will give qemu a
filename and recieve from qemu the size/contents of that file. qemu will do
read/writes on the host to satisfy such requires. This is not an argument
against adding file sharing network support in qemu though.

The way I see it:

OSes such as DOS, which are well known and still semi-popular can use their own
qemu guest app to handle this without the difficulty of setting up networking
(sidenote: i've never heard of anyone getting networking to work under DOS in
qemu).

OSes such as Minix, which are not as well known nor as popular but support
networking easily can just use the tftp server (tho minix itself seems to
lack a tftp client) without the need of having to write a special guest app.

-- 
Infinite complexity begets infinite beauty.
Infinite precision begets infinite perfection.