support git+mosh for unreliable connections

support git+mosh for unreliable connections

[Pirate Praveen]

[-- Attachment #1: Type: text/plain, Size: 602 bytes --]

Hi,

 When working with big projects over a slow, unreliable connection,
currently there is no way to resume a clone or pull when the connection
breaks. mosh is a better replacement for ssh over unreliable
connections. supporting git+mosh protocol will go a long way in
supporting people who work with unreliable, mobile networks, especially
in developed countries (I personally have to try many times when working
with large projects as my 3g mobile connection keeps dropping. I
recently discovered mosh and it works like a charm. More about mosh
https://mosh.mit.edu/

Thanks
Praveen


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: support git+mosh for unreliable connections

[Dennis Kaarsemaker]

On wo, 2015-04-15 at 18:37 +0530, Pirate Praveen wrote:
> Hi,
> 
>  When working with big projects over a slow, unreliable connection,
> currently there is no way to resume a clone or pull when the connection
> breaks. mosh is a better replacement for ssh over unreliable
> connections. supporting git+mosh protocol will go a long way in
> supporting people who work with unreliable, mobile networks, especially
> in developed countries (I personally have to try many times when working
> with large projects as my 3g mobile connection keeps dropping. I
> recently discovered mosh and it works like a charm. More about mosh
> https://mosh.mit.edu/

mosh isn't a generic transport though, it's a udp-based session state
synchronization protocol. I don't think it can be used as a git
transport.

-- 
Dennis Kaarsemaker
www.kaarsemaker.net

Re: support git+mosh for unreliable connections

[Michael J Gruber]

Pirate Praveen venit, vidit, dixit 15.04.2015 15:07:
> Hi,
> 
>  When working with big projects over a slow, unreliable connection,
> currently there is no way to resume a clone or pull when the connection
> breaks. mosh is a better replacement for ssh over unreliable
> connections. supporting git+mosh protocol will go a long way in
> supporting people who work with unreliable, mobile networks, especially
> in developed countries (I personally have to try many times when working
> with large projects as my 3g mobile connection keeps dropping. I
> recently discovered mosh and it works like a charm. More about mosh
> https://mosh.mit.edu/

What would that require git to do, beyond taking whatever you tell it
(using GIT_SSH or _GIT_SSH_COMMAND) to use as a drop in replacement for ssh?

Michael

Re: support git+mosh for unreliable connections

[Pirate Praveen]

[-- Attachment #1: Type: text/plain, Size: 841 bytes --]

On Wednesday 15 April 2015 07:22 PM, Michael J Gruber wrote:
> What would that require git to do, beyond taking whatever you tell it
> (using GIT_SSH or _GIT_SSH_COMMAND) to use as a drop in replacement for ssh?
> 
> Michael
> 

May be support git+mosh as a protocol, since it is not a drop in
replacement. It is redesigned remote shell. The ideas it uses for
session resumption needs to be reimplemented. This will need support
from git, because it needs server side to be modified. Use SSP to return
the the current progress for a particular session (it uses AES session ids).

So when a client connect with a session id, git server side can respond
with the current state, how many objects received in that session, and
client can continue from where it stopped. Client also will need to
store session information.




[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: support git+mosh for unreliable connections

[Johannes Schindelin]

Hi Praveen,

On 2015-04-15 16:18, Pirate Praveen wrote:
> On Wednesday 15 April 2015 07:22 PM, Michael J Gruber wrote:
>> What would that require git to do, beyond taking whatever you tell it
>> (using GIT_SSH or _GIT_SSH_COMMAND) to use as a drop in replacement for ssh?
> 
> May be support git+mosh as a protocol, since it is not a drop in
> replacement. It is redesigned remote shell. The ideas it uses for
> session resumption needs to be reimplemented. This will need support
> from git, because it needs server side to be modified. Use SSP to return
> the the current progress for a particular session (it uses AES session ids).

It will need support from Git alright, but not as much as from mosh, see my other reply: Mosh was not designed for non-interactive use. That support would have to be added before we can go any further.

> So when a client connect with a session id, git server side can respond
> with the current state, how many objects received in that session, and
> client can continue from where it stopped. Client also will need to
> store session information.

No, the protocol can stay exactly the same, once you have a way to communicate non-interactively via mosh.

Ciao,
Johannes

Re: support git+mosh for unreliable connections

[Trevor Saunders]

On Wed, Apr 15, 2015 at 04:41:42PM +0200, Johannes Schindelin wrote:
> Hi Praveen,
> 
> On 2015-04-15 16:18, Pirate Praveen wrote:
> > On Wednesday 15 April 2015 07:22 PM, Michael J Gruber wrote:
> >> What would that require git to do, beyond taking whatever you tell it
> >> (using GIT_SSH or _GIT_SSH_COMMAND) to use as a drop in replacement for ssh?
> > 
> > May be support git+mosh as a protocol, since it is not a drop in
> > replacement. It is redesigned remote shell. The ideas it uses for
> > session resumption needs to be reimplemented. This will need support
> > from git, because it needs server side to be modified. Use SSP to return
> > the the current progress for a particular session (it uses AES session ids).
> 
> It will need support from Git alright, but not as much as from mosh, see my other reply: Mosh was not designed for non-interactive use. That support would have to be added before we can go any further.

is that really relevent? mosh doesn't support things like X forwarding
or port forwarding, but it certainly does support ssh <host> <command>
and then doing IO.  It might not support not doing terminal emulation
stuff, but that seems like a simple thing to change in principal at which
point I think it would support enough of ssh's functionality its a drop
in replacement as far as git is concerned.  Seems to me mosh is close
enough on its own its worth experimentation by someone who cares.

Trev

> > So when a client connect with a session id, git server side can respond
> > with the current state, how many objects received in that session, and
> > client can continue from where it stopped. Client also will need to
> > store session information.
> 
> No, the protocol can stay exactly the same, once you have a way to communicate non-interactively via mosh.
> 
> Ciao,
> Johannes
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: support git+mosh for unreliable connections

[Johannes Schindelin]

Hi Trevor,

On 2015-04-15 17:33, Trevor Saunders wrote:
> On Wed, Apr 15, 2015 at 04:41:42PM +0200, Johannes Schindelin wrote:
>>
>> On 2015-04-15 16:18, Pirate Praveen wrote:
>> > On Wednesday 15 April 2015 07:22 PM, Michael J Gruber wrote:
>> >> What would that require git to do, beyond taking whatever you tell it
>> >> (using GIT_SSH or _GIT_SSH_COMMAND) to use as a drop in replacement for ssh?
>> >
>> > May be support git+mosh as a protocol, since it is not a drop in
>> > replacement. It is redesigned remote shell. The ideas it uses for
>> > session resumption needs to be reimplemented. This will need support
>> > from git, because it needs server side to be modified. Use SSP to return
>> > the the current progress for a particular session (it uses AES session ids).
>>
>> It will need support from Git alright, but not as much as from mosh, see my other reply: Mosh was not designed for non-interactive use. That support would have to be added before we can go any further.
> 
> is that really relevent? mosh doesn't support things like X forwarding
> or port forwarding, but it certainly does support ssh <host> <command>
> and then doing IO.

Ah, so mosh's README lied to me!

If `mosh <user>@<host> <command>` works, then a simple `GIT_SSH=mosh` should work out of the box, too. Have you tried it?

Ciao,
Johannes
  It might not support not doing terminal emulation
> stuff, but that seems like a simple thing to change in principal at which
> point I think it would support enough of ssh's functionality its a drop
> in replacement as far as git is concerned.  Seems to me mosh is close
> enough on its own its worth experimentation by someone who cares.
> 
> Trev
> 
>> > So when a client connect with a session id, git server side can respond
>> > with the current state, how many objects received in that session, and
>> > client can continue from where it stopped. Client also will need to
>> > store session information.
>>
>> No, the protocol can stay exactly the same, once you have a way to communicate non-interactively via mosh.
>>
>> Ciao,
>> Johannes
>> --
>> To unsubscribe from this list: send the line "unsubscribe git" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: support git+mosh for unreliable connections

[Trevor Saunders]

On Wed, Apr 15, 2015 at 07:46:15PM +0200, Johannes Schindelin wrote:
> Hi Trevor,
> 
> On 2015-04-15 17:33, Trevor Saunders wrote:
> > On Wed, Apr 15, 2015 at 04:41:42PM +0200, Johannes Schindelin wrote:
> >>
> >> On 2015-04-15 16:18, Pirate Praveen wrote:
> >> > On Wednesday 15 April 2015 07:22 PM, Michael J Gruber wrote:
> >> >> What would that require git to do, beyond taking whatever you tell it
> >> >> (using GIT_SSH or _GIT_SSH_COMMAND) to use as a drop in replacement for ssh?
> >> >
> >> > May be support git+mosh as a protocol, since it is not a drop in
> >> > replacement. It is redesigned remote shell. The ideas it uses for
> >> > session resumption needs to be reimplemented. This will need support
> >> > from git, because it needs server side to be modified. Use SSP to return
> >> > the the current progress for a particular session (it uses AES session ids).
> >>
> >> It will need support from Git alright, but not as much as from mosh, see my other reply: Mosh was not designed for non-interactive use. That support would have to be added before we can go any further.
> > 
> > is that really relevent? mosh doesn't support things like X forwarding
> > or port forwarding, but it certainly does support ssh <host> <command>
> > and then doing IO.
> 
> Ah, so mosh's README lied to me!

I wouldn't say it lied, its just not really clear what is "interactive"
I'd say git's use of ssh is kind of interactive compared to things like
port forwarding.

> If `mosh <user>@<host> <command>` works, then a simple `GIT_SSH=mosh` should work out of the box, too. Have you tried it?

it does work, I just tried mosh $host cat and then typing stuff and
having it printed back at me.  However it clears the terminal before
hand and prints a message on exit.  I tried GIT_SSH=mosh git clone and
it failed, but I haven't really dug into why.  SO I suspect this can be
made to work with some work on the mosh side, but I'm not sure exactly
how ssh and mosh are behaving differently here.

Trev

> 
> Ciao,
> Johannes
>   It might not support not doing terminal emulation
> > stuff, but that seems like a simple thing to change in principal at which
> > point I think it would support enough of ssh's functionality its a drop
> > in replacement as far as git is concerned.  Seems to me mosh is close
> > enough on its own its worth experimentation by someone who cares.
> > 
> > Trev
> > 
> >> > So when a client connect with a session id, git server side can respond
> >> > with the current state, how many objects received in that session, and
> >> > client can continue from where it stopped. Client also will need to
> >> > store session information.
> >>
> >> No, the protocol can stay exactly the same, once you have a way to communicate non-interactively via mosh.
> >>
> >> Ciao,
> >> Johannes
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe git" in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > --
> > To unsubscribe from this list: send the line "unsubscribe git" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: support git+mosh for unreliable connections

[Michael J Gruber]

Trevor Saunders venit, vidit, dixit 15.04.2015 20:59:
> On Wed, Apr 15, 2015 at 07:46:15PM +0200, Johannes Schindelin wrote:
>> Hi Trevor,
>>
>> On 2015-04-15 17:33, Trevor Saunders wrote:
>>> On Wed, Apr 15, 2015 at 04:41:42PM +0200, Johannes Schindelin wrote:
>>>>
>>>> On 2015-04-15 16:18, Pirate Praveen wrote:
>>>>> On Wednesday 15 April 2015 07:22 PM, Michael J Gruber wrote:
>>>>>> What would that require git to do, beyond taking whatever you tell it
>>>>>> (using GIT_SSH or _GIT_SSH_COMMAND) to use as a drop in replacement for ssh?
>>>>>
>>>>> May be support git+mosh as a protocol, since it is not a drop in
>>>>> replacement. It is redesigned remote shell. The ideas it uses for
>>>>> session resumption needs to be reimplemented. This will need support
>>>>> from git, because it needs server side to be modified. Use SSP to return
>>>>> the the current progress for a particular session (it uses AES session ids).
>>>>
>>>> It will need support from Git alright, but not as much as from mosh, see my other reply: Mosh was not designed for non-interactive use. That support would have to be added before we can go any further.
>>>
>>> is that really relevent? mosh doesn't support things like X forwarding
>>> or port forwarding, but it certainly does support ssh <host> <command>
>>> and then doing IO.
>>
>> Ah, so mosh's README lied to me!
> 
> I wouldn't say it lied, its just not really clear what is "interactive"
> I'd say git's use of ssh is kind of interactive compared to things like
> port forwarding.
> 
>> If `mosh <user>@<host> <command>` works, then a simple `GIT_SSH=mosh` should work out of the box, too. Have you tried it?
> 
> it does work, I just tried mosh $host cat and then typing stuff and
> having it printed back at me.  However it clears the terminal before
> hand and prints a message on exit.  I tried GIT_SSH=mosh git clone and
> it failed, but I haven't really dug into why.  SO I suspect this can be
> made to work with some work on the mosh side, but I'm not sure exactly
> how ssh and mosh are behaving differently here.
> 
> Trev

First thing you see on mosh.mit.edu:

"Mosh is a replacement for SSH."

I guess that needs a footnote...

Michael

Re: support git+mosh for unreliable connections

[Dennis Kaarsemaker]

On wo, 2015-04-15 at 19:46 +0200, Johannes Schindelin wrote:
> On 2015-04-15 17:33, Trevor Saunders wrote:

> > but it certainly does support ssh <host> <command>
> > and then doing IO.
> 
Yes, in interactive sessions. mosh synchronizes terminal state, it
doesn't allow random I/O between client and server.

> Ah, so mosh's README lied to me!
> 
> If `mosh <user>@<host> <command>` works, then a simple `GIT_SSH=mosh`
> should work out of the box, too. Have you tried it?

It does not and cannot work. The way mosh works, is that it uses ssh to
log in and launch a mosh-server daemon. This daemon and the mosh client
then communicate via a custom UDP protocol. The SSH connection is closed
after the mosh-server has been launched as it is no longer needed.

The communication between the mosh client and server synchronizes
terminal state, somewhat like what screen/tmux do.

-- 
Dennis Kaarsemaker
www.kaarsemaker.net

Re: support git+mosh for unreliable connections

[Andreas Krey]

On Wed, 15 Apr 2015 21:25:44 +0000, Dennis Kaarsemaker wrote:
...
> It does not and cannot work. The way mosh works, is that it uses ssh to
> log in and launch a mosh-server daemon. This daemon and the mosh client
> then communicate via a custom UDP protocol. The SSH connection is closed
> after the mosh-server has been launched as it is no longer needed.
> 
> The communication between the mosh client and server synchronizes
> terminal state, somewhat like what screen/tmux do.

I object to the 'can not' part a bit. There is (1) the terminal state
prediction and (2) the reliable-over-reconnects communication, and for
a noninteractive usage you'd need only (2).

Once upon a time I implemented a simple UDP server and client;
the client to be used as a ProxyCommand in ssh, and the server
just talks to the local ssh server. This pretty much does what
the OP wants, and it works just as a transport for ssh, so
all ssh features are there (but of course there is no
terminal prediction). Unfortunately it needs to be ported
to libev/libuv before it could be released. It's *much*
simpler than mosh, although the use-ssh-to-start-server
trick would be nice.)

Andreas

-- 
"Totally trivial. Famous last words."
From: Linus Torvalds <torvalds@*.org>
Date: Fri, 22 Jan 2010 07:29:21 -0800

Re: support git+mosh for unreliable connections

[Johannes Schindelin]

Hi Praveen "Arrrr",

On 2015-04-15 15:07, Pirate Praveen wrote:

>  When working with big projects over a slow, unreliable connection,
> currently there is no way to resume a clone or pull when the connection
> breaks. mosh is a better replacement for ssh over unreliable
> connections. supporting git+mosh protocol will go a long way in
> supporting people who work with unreliable, mobile networks, especially
> in developed countries (I personally have to try many times when working
> with large projects as my 3g mobile connection keeps dropping. I
> recently discovered mosh and it works like a charm. More about mosh
> https://mosh.mit.edu/

>From https://github.com/keithw/mosh:

> Mosh does not support X forwarding or the non-interactive uses of SSH, including port forwarding.

In particular it "does not support [...] the non-interactive uses of SSH", which the git+mosh transport would require, though.

That means that you would have to invest quite a bit of effort into enhancing mosh to *support* the non-interactive uses of SSH before you could start implementing `git-remote-mosh`...

Ciao,
Johannes

Re: support git+mosh for unreliable connections

[Pirate Praveen]

[-- Attachment #1: Type: text/plain, Size: 1121 bytes --]

On Wednesday 15 April 2015 07:52 PM, Johannes Schindelin wrote:
> From https://github.com/keithw/mosh:
> 
>> Mosh does not support X forwarding or the non-interactive uses of SSH, including port forwarding.
> 
> In particular it "does not support [...] the non-interactive uses of SSH", which the git+mosh transport would require, though.
> 
> That means that you would have to invest quite a bit of effort into enhancing mosh to *support* the non-interactive uses of SSH before you could start implementing `git-remote-mosh`...
> 
> Ciao,
> Johannes
> 

Q: Are the mosh principles relevant to other network applications?

    We think so. The design principles that Mosh stands for are
conservative: warning the user if the state being displayed is out of
date, serializing and checkpointing all transactions so that if there
are no warnings, the user knows every prior transaction has succeeded,
and handling expected events (like roaming from one WiFi network to
another) gracefully.

Can the ideas be used to resume a pull, push or clone operation?
Especially serializing and checkpointing.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: support git+mosh for unreliable connections

[Ilari Liusvaara]

On Wed, Apr 15, 2015 at 08:13:51PM +0530, Pirate Praveen wrote:
> 
> Q: Are the mosh principles relevant to other network applications?
> 
>     We think so. The design principles that Mosh stands for are
> conservative: warning the user if the state being displayed is out of
> date, serializing and checkpointing all transactions so that if there
> are no warnings, the user knows every prior transaction has succeeded,
> and handling expected events (like roaming from one WiFi network to
> another) gracefully.
> 
> Can the ideas be used to resume a pull, push or clone operation?
> Especially serializing and checkpointing.

Well, it is possible to write a remote helper and serverside program
that internally handles connection unreliability, so Git itself
(upload-archive, upload-pack, receive-pack, archive, fetch-pack
and send-pack) sees a reliable (full-duplex, half-closeable, stream)
channel.

Suitably done, that can "resume" (from Git POV, nothing special
happened) across things like IP address changes.

However, that is quite difficult to do in practice. Not because
interface to Git is complicated, but because the transport problem
itself is complicated (however, it still seems way easier than
making Git internally be able to resume interrupted operations).

Mosh needs to solve at least most of that, it just doesn't provode
the right kind of interface.


-Ilari

Re: support git+mosh for unreliable connections

[Pirate Praveen]

[-- Attachment #1: Type: text/plain, Size: 1608 bytes --]

On Thursday 16 April 2015 01:56 AM, Ilari Liusvaara wrote:
> On Wed, Apr 15, 2015 at 08:13:51PM +0530, Pirate Praveen wrote:
>>
>> Q: Are the mosh principles relevant to other network applications?
>>
>>     We think so. The design principles that Mosh stands for are
>> conservative: warning the user if the state being displayed is out of
>> date, serializing and checkpointing all transactions so that if there
>> are no warnings, the user knows every prior transaction has succeeded,
>> and handling expected events (like roaming from one WiFi network to
>> another) gracefully.
>>
>> Can the ideas be used to resume a pull, push or clone operation?
>> Especially serializing and checkpointing.
> 
> Well, it is possible to write a remote helper and serverside program
> that internally handles connection unreliability, so Git itself
> (upload-archive, upload-pack, receive-pack, archive, fetch-pack
> and send-pack) sees a reliable (full-duplex, half-closeable, stream)
> channel.
> 
> Suitably done, that can "resume" (from Git POV, nothing special
> happened) across things like IP address changes.
> 
> However, that is quite difficult to do in practice. Not because
> interface to Git is complicated, but because the transport problem
> itself is complicated (however, it still seems way easier than
> making Git internally be able to resume interrupted operations).
> 
> Mosh needs to solve at least most of that, it just doesn't provode
> the right kind of interface.

I have requested mosh team to fix these issues
https://github.com/keithw/mosh/issues/597



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]