[RFC] non-preemptible kernel socket for RAMster

[RFC] non-preemptible kernel socket for RAMster

[Dan Magenheimer]

In working on a kernel project called RAMster* (where RAM on a
remote system may be used for clean page cache pages and for swap
pages), I found I have need for a kernel socket to be used when
in non-preemptible state.  I admit to being a networking idiot,
but I have been successfully using the following small patch.
I'm not sure whether I am lucky so far... perhaps more
sockets or larger/different loads will require a lot more
changes (or maybe even make my objective impossible).
So I thought I'd post it for comment.  I'd appreciate
any thoughts or suggestions.

Thanks,
Dan

* http://events.linuxfoundation.org/events/linuxcon/magenheimer 

diff -Napur linux-2.6.37/net/core/sock.c linux-2.6.37-ramster/net/core/sock.c
--- linux-2.6.37/net/core/sock.c	2011-07-03 19:14:52.267853088 -0600
+++ linux-2.6.37-ramster/net/core/sock.c	2011-07-03 19:10:04.340980799 -0600
@@ -1587,6 +1587,14 @@ static void __lock_sock(struct sock *sk)
 	__acquires(&sk->sk_lock.slock)
 {
 	DEFINE_WAIT(wait);
+	if (!preemptible()) {
+		while (sock_owned_by_user(sk)) {
+			spin_unlock_bh(&sk->sk_lock.slock);
+			cpu_relax();
+			spin_lock_bh(&sk->sk_lock.slock);
+		}
+		return;
+	}
 
 	for (;;) {
 		prepare_to_wait_exclusive(&sk->sk_lock.wq, &wait,
@@ -1623,7 +1631,8 @@ static void __release_sock(struct sock *
 			 * This is safe to do because we've taken the backlog
 			 * queue private:
 			 */
-			cond_resched_softirq();
+			if (preemptible())
+				cond_resched_softirq();
 			skb = next;
 		} while (skb != NULL);

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC] non-preemptible kernel socket for RAMster

[Eric Dumazet]

Le mardi 05 juillet 2011 A  08:54 -0700, Dan Magenheimer a A(C)crit :
> In working on a kernel project called RAMster* (where RAM on a
> remote system may be used for clean page cache pages and for swap
> pages), I found I have need for a kernel socket to be used when
> in non-preemptible state.  I admit to being a networking idiot,
> but I have been successfully using the following small patch.
> I'm not sure whether I am lucky so far... perhaps more
> sockets or larger/different loads will require a lot more
> changes (or maybe even make my objective impossible).
> So I thought I'd post it for comment.  I'd appreciate
> any thoughts or suggestions.
> 
> Thanks,
> Dan
> 
> * http://events.linuxfoundation.org/events/linuxcon/magenheimer 
> 
> diff -Napur linux-2.6.37/net/core/sock.c linux-2.6.37-ramster/net/core/sock.c
> --- linux-2.6.37/net/core/sock.c	2011-07-03 19:14:52.267853088 -0600
> +++ linux-2.6.37-ramster/net/core/sock.c	2011-07-03 19:10:04.340980799 -0600
> @@ -1587,6 +1587,14 @@ static void __lock_sock(struct sock *sk)
>  	__acquires(&sk->sk_lock.slock)
>  {
>  	DEFINE_WAIT(wait);
> +	if (!preemptible()) {
> +		while (sock_owned_by_user(sk)) {
> +			spin_unlock_bh(&sk->sk_lock.slock);
> +			cpu_relax();
> +			spin_lock_bh(&sk->sk_lock.slock);
> +		}
> +		return;
> +	}

Hmm, was this tested on UP machine ?

>  
>  	for (;;) {
>  		prepare_to_wait_exclusive(&sk->sk_lock.wq, &wait,
> @@ -1623,7 +1631,8 @@ static void __release_sock(struct sock *
>  			 * This is safe to do because we've taken the backlog
>  			 * queue private:
>  			 */
> -			cond_resched_softirq();
> +			if (preemptible())
> +				cond_resched_softirq();
>  			skb = next;
>  		} while (skb != NULL);


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

RE: [RFC] non-preemptible kernel socket for RAMster

[Dan Magenheimer]

> From: Eric Dumazet [mailto:eric.dumazet@gmail.com]
> Sent: Tuesday, July 05, 2011 10:31 AM
> To: Dan Magenheimer
> Cc: netdev@vger.kernel.org; Konrad Wilk; linux-mm
> Subject: Re: [RFC] non-preemptible kernel socket for RAMster
> 
> Le mardi 05 juillet 2011 à 08:54 -0700, Dan Magenheimer a écrit :
> > In working on a kernel project called RAMster* (where RAM on a
> > remote system may be used for clean page cache pages and for swap
> > pages), I found I have need for a kernel socket to be used when
> > in non-preemptible state.  I admit to being a networking idiot,
> > but I have been successfully using the following small patch.
> > I'm not sure whether I am lucky so far... perhaps more
> > sockets or larger/different loads will require a lot more
> > changes (or maybe even make my objective impossible).
> > So I thought I'd post it for comment.  I'd appreciate
> > any thoughts or suggestions.
> >
> > Thanks,
> > Dan
> >
> > * http://events.linuxfoundation.org/events/linuxcon/magenheimer
> >
> > diff -Napur linux-2.6.37/net/core/sock.c linux-2.6.37-ramster/net/core/sock.c
> > --- linux-2.6.37/net/core/sock.c	2011-07-03 19:14:52.267853088 -0600
> > +++ linux-2.6.37-ramster/net/core/sock.c	2011-07-03 19:10:04.340980799 -0600
> > @@ -1587,6 +1587,14 @@ static void __lock_sock(struct sock *sk)
> >  	__acquires(&sk->sk_lock.slock)
> >  {
> >  	DEFINE_WAIT(wait);
> > +	if (!preemptible()) {
> > +		while (sock_owned_by_user(sk)) {
> > +			spin_unlock_bh(&sk->sk_lock.slock);
> > +			cpu_relax();
> > +			spin_lock_bh(&sk->sk_lock.slock);
> > +		}
> > +		return;
> > +	}
> 
> Hmm, was this tested on UP machine ?

Hi Eric --

Thanks for the reply!

I hadn't tested UP in awhile so am testing now, and it seems to
work OK so far.  However, I am just testing my socket, *not* testing
sockets in general.  Are you implying that this patch will
break (kernel) sockets in general on a UP machine?  If so,
could you be more specific as to why?  (Again, I said
I am a networking idiot. ;-)  I played a bit with adding
a new SOCK_ flag and triggering off of that, but this
version of the patch seemed much simpler.

Thanks,
Dan

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

RE: [RFC] non-preemptible kernel socket for RAMster

[Eric Dumazet]

Le mardi 05 juillet 2011 A  10:25 -0700, Dan Magenheimer a A(C)crit :
> > From: Eric Dumazet [mailto:eric.dumazet@gmail.com]
> > Sent: Tuesday, July 05, 2011 10:31 AM
> > To: Dan Magenheimer
> > Cc: netdev@vger.kernel.org; Konrad Wilk; linux-mm
> > Subject: Re: [RFC] non-preemptible kernel socket for RAMster
> > 
> > Le mardi 05 juillet 2011 A  08:54 -0700, Dan Magenheimer a A(C)crit :
> > > In working on a kernel project called RAMster* (where RAM on a
> > > remote system may be used for clean page cache pages and for swap
> > > pages), I found I have need for a kernel socket to be used when
> > > in non-preemptible state.  I admit to being a networking idiot,
> > > but I have been successfully using the following small patch.
> > > I'm not sure whether I am lucky so far... perhaps more
> > > sockets or larger/different loads will require a lot more
> > > changes (or maybe even make my objective impossible).
> > > So I thought I'd post it for comment.  I'd appreciate
> > > any thoughts or suggestions.
> > >
> > > Thanks,
> > > Dan
> > >
> > > * http://events.linuxfoundation.org/events/linuxcon/magenheimer
> > >
> > > diff -Napur linux-2.6.37/net/core/sock.c linux-2.6.37-ramster/net/core/sock.c
> > > --- linux-2.6.37/net/core/sock.c	2011-07-03 19:14:52.267853088 -0600
> > > +++ linux-2.6.37-ramster/net/core/sock.c	2011-07-03 19:10:04.340980799 -0600
> > > @@ -1587,6 +1587,14 @@ static void __lock_sock(struct sock *sk)
> > >  	__acquires(&sk->sk_lock.slock)
> > >  {
> > >  	DEFINE_WAIT(wait);
> > > +	if (!preemptible()) {
> > > +		while (sock_owned_by_user(sk)) {
> > > +			spin_unlock_bh(&sk->sk_lock.slock);
> > > +			cpu_relax();
> > > +			spin_lock_bh(&sk->sk_lock.slock);
> > > +		}
> > > +		return;
> > > +	}
> > 
> > Hmm, was this tested on UP machine ?
> 
> Hi Eric --
> 
> Thanks for the reply!
> 
> I hadn't tested UP in awhile so am testing now, and it seems to
> work OK so far.  However, I am just testing my socket, *not* testing
> sockets in general.  Are you implying that this patch will
> break (kernel) sockets in general on a UP machine?  If so,
> could you be more specific as to why?  (Again, I said
> I am a networking idiot. ;-)  I played a bit with adding
> a new SOCK_ flag and triggering off of that, but this
> version of the patch seemed much simpler.

Say you have two processes and socket S

One process locks socket S, and is preempted by another process.

This second process is non preemptible and try to lock same socket.

-> deadlock, since P1 never releases socket S



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

RE: [RFC] non-preemptible kernel socket for RAMster

[Dan Magenheimer]

> > > > +++ linux-2.6.37-ramster/net/core/sock.c	2011-07-03 19:10:04.340980799 -0600
> > > > @@ -1587,6 +1587,14 @@ static void __lock_sock(struct sock *sk)
> > > >  	__acquires(&sk->sk_lock.slock)
> > > >  {
> > > >  	DEFINE_WAIT(wait);
> > > > +	if (!preemptible()) {
> > > > +		while (sock_owned_by_user(sk)) {
> > > > +			spin_unlock_bh(&sk->sk_lock.slock);
> > > > +			cpu_relax();
> > > > +			spin_lock_bh(&sk->sk_lock.slock);
> > > > +		}
> > > > +		return;
> > > > +	}
> > >
> > > Hmm, was this tested on UP machine ?
> >
> > Hi Eric --
> >
> > Thanks for the reply!
> >
> > I hadn't tested UP in awhile so am testing now, and it seems to
> > work OK so far.  However, I am just testing my socket, *not* testing
> > sockets in general.  Are you implying that this patch will
> > break (kernel) sockets in general on a UP machine?  If so,
> > could you be more specific as to why?  (Again, I said
> > I am a networking idiot. ;-)  I played a bit with adding
> > a new SOCK_ flag and triggering off of that, but this
> > version of the patch seemed much simpler.
> 
> Say you have two processes and socket S
> 
> One process locks socket S, and is preempted by another process.
> 
> This second process is non preemptible and try to lock same socket.
> 
> -> deadlock, since P1 never releases socket S

Oh, OK.  My use model is that a socket that is used non-preemptible
must always be used non-preemptible.  In other words, this kind
of socket is an extreme form of non-blocking.  Doesn't that seem
like a reasonable constraint? 

Thanks,
Dan

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

RE: [RFC] non-preemptible kernel socket for RAMster

[Loke, Chetan]

> -----Original Message-----
> From: netdev-owner@vger.kernel.org [mailto:netdev-
> owner@vger.kernel.org] On Behalf Of Dan Magenheimer
> Sent: July 05, 2011 11:54 AM
> To: netdev@vger.kernel.org
> Cc: Konrad Wilk; linux-mm
> Subject: [RFC] non-preemptible kernel socket for RAMster
> 
> In working on a kernel project called RAMster* (where RAM on a
> remote system may be used for clean page cache pages and for swap
> pages), I found I have need for a kernel socket to be used when


How is RAMster+swap different than NBD's (pending etc?)support for SWAP
over NBD?


Chetan Loke

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

RE: [RFC] non-preemptible kernel socket for RAMster

[Dan Magenheimer]

> From: Loke, Chetan [mailto:Chetan.Loke@netscout.com]
> Sent: Tuesday, July 05, 2011 10:37 AM
> To: Dan Magenheimer; netdev@vger.kernel.org
> Cc: Konrad Wilk; linux-mm
> Subject: RE: [RFC] non-preemptible kernel socket for RAMster
> 
> > In working on a kernel project called RAMster* (where RAM on a
> > remote system may be used for clean page cache pages and for swap
> > pages), I found I have need for a kernel socket to be used when
> 
> How is RAMster+swap different than NBD's (pending etc?)support for SWAP
> over NBD?

Hi Chetan --

Thanks for your question.

I may be ignorant of details about NBD, but did some quick
research using google.  If I understand correctly, swap over
NBD is still writing to a configured swap disk on the remote
machine.  RAMster is swapping to *RAM* on the remote machine.
The idea is that most machines are very overprovisioned in
RAM, and are rarely using all of their RAM, especially when
a machine is (mostly) idle.  In other words, the "max of
the sums" of RAM usage on a group of machines is much lower
than the "sum of the max" of RAM usage.

So if the network is sufficiently faster than disk for
moving a page of data, RAMster provides a significant
performance improvement.  OR RAMster may allow a significant
reduction in the total amount of RAM across a data center.

The version of RAMster I am working on now is really
a proof-of-concept that works over sockets, using the
ocfs2 cluster layer.  One can easily envision a future
"exo-fabric" which allows one machine to write to the
RAM of another machine... for this future hardware,
RAMster becomes much more interesting.

Thanks,
Dan

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

RE: [RFC] non-preemptible kernel socket for RAMster

[Loke, Chetan]

> -----Original Message-----
> From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
> Sent: July 05, 2011 1:25 PM
> To: Loke, Chetan; netdev@vger.kernel.org
> Cc: Konrad Wilk; linux-mm
> Subject: RE: [RFC] non-preemptible kernel socket for RAMster
> 
> > From: Loke, Chetan [mailto:Chetan.Loke@netscout.com]
> > Sent: Tuesday, July 05, 2011 10:37 AM
> > To: Dan Magenheimer; netdev@vger.kernel.org
> > Cc: Konrad Wilk; linux-mm
> > Subject: RE: [RFC] non-preemptible kernel socket for RAMster
> >
> > > In working on a kernel project called RAMster* (where RAM on a
> > > remote system may be used for clean page cache pages and for swap
> > > pages), I found I have need for a kernel socket to be used when
> >
> > How is RAMster+swap different than NBD's (pending etc?)support for
> SWAP
> > over NBD?
> 
> Hi Chetan --
> 
> Thanks for your question.
> 
> I may be ignorant of details about NBD, but did some quick
> research using google.  If I understand correctly, swap over
> NBD is still writing to a configured swap disk on the remote

Hi - I thought NBD-server needs a backing store(a file). 
Now the file itself could reside on a RAM-drive or disk-drive etc.
And so a remote NBD(disk or RAM) can be mounted locally as a swap
device.
The local client should still see it as a block device.

I haven't used the RAM-drive feature myself but you may want to check if
it
works or even borrow that logic in your code.


> machine.  RAMster is swapping to *RAM* on the remote machine.
> The idea is that most machines are very overprovisioned in
> RAM, and are rarely using all of their RAM, especially when
> a machine is (mostly) idle.  In other words, the "max of
> the sums" of RAM usage on a group of machines is much lower
> than the "sum of the max" of RAM usage.
> 
> So if the network is sufficiently faster than disk for
> moving a page of data, RAMster provides a significant
> performance improvement.  OR RAMster may allow a significant
> reduction in the total amount of RAM across a data center.
> 
> The version of RAMster I am working on now is really
> a proof-of-concept that works over sockets, using the
> ocfs2 cluster layer.  One can easily envision a future
> "exo-fabric" which allows one machine to write to the
> RAM of another machine... for this future hardware,
> RAMster becomes much more interesting.
> 

Or you can also try scst-in-RAM mode(if you want to experiment with
different fabrics).


> Thanks,
> Dan

Thanks
Chetan Loke

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

RE: [RFC] non-preemptible kernel socket for RAMster

[Dan Magenheimer]

> From: Loke, Chetan [mailto:Chetan.Loke@netscout.com]
> > From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
> > Subject: RE: [RFC] non-preemptible kernel socket for RAMster
> >
> > > From: Loke, Chetan [mailto:Chetan.Loke@netscout.com]
> > > Sent: Tuesday, July 05, 2011 10:37 AM
> > > To: Dan Magenheimer; netdev@vger.kernel.org
> > > Cc: Konrad Wilk; linux-mm
> > > Subject: RE: [RFC] non-preemptible kernel socket for RAMster
> > >
> > > > In working on a kernel project called RAMster* (where RAM on a
> > > > remote system may be used for clean page cache pages and for swap
> > > > pages), I found I have need for a kernel socket to be used when
> > >
> > > How is RAMster+swap different than NBD's (pending etc?)support for
> > > SWAP over NBD?
> >
> > I may be ignorant of details about NBD, but did some quick
> > research using google.  If I understand correctly, swap over
> > NBD is still writing to a configured swap disk on the remote
> 
> Hi - I thought NBD-server needs a backing store(a file).
> Now the file itself could reside on a RAM-drive or disk-drive etc.
> And so a remote NBD(disk or RAM) can be mounted locally as a swap
> device.
> The local client should still see it as a block device.
> 
> I haven't used the RAM-drive feature myself but you may want to check if
> it
> works or even borrow that logic in your code.

Actually, RAMster is using a much more flexible type of
RAM-drive; it is built on top of Transcendent Memory
and on top of zcache (and thus on top of cleancache and
frontswap).  A RAM-drive is fixed size so is not very suitable
for the flexibility required for RAMster.  For example,
suppose you have two machines A and B.  At one point in
time A is overcommitted and needs to swap and B is relatively
idle.  Then later, B is overcommitted and needs to swap and
A is relatively idle.  RAMster can handle this entirely
dynamically, a RAM-drive cannot.

> > machine.  RAMster is swapping to *RAM* on the remote machine.
> > The idea is that most machines are very overprovisioned in
> > RAM, and are rarely using all of their RAM, especially when
> > a machine is (mostly) idle.  In other words, the "max of
> > the sums" of RAM usage on a group of machines is much lower
> > than the "sum of the max" of RAM usage.
> >
> > So if the network is sufficiently faster than disk for
> > moving a page of data, RAMster provides a significant
> > performance improvement.  OR RAMster may allow a significant
> > reduction in the total amount of RAM across a data center.
> >
> > The version of RAMster I am working on now is really
> > a proof-of-concept that works over sockets, using the
> > ocfs2 cluster layer.  One can easily envision a future
> > "exo-fabric" which allows one machine to write to the
> > RAM of another machine... for this future hardware,
> > RAMster becomes much more interesting.
> 
> Or you can also try scst-in-RAM mode(if you want to experiment with
> different fabrics).

Thanks.  Could you provide a pointer for this?  I found
the SCST sourceforge page but no obvious references to
scst-in-ram-mode.  (But also, since it appears to be
SCSI-related, I wonder if it also assumes a fixed size
target device, RAM or disk or ??)

Dan

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

RE: [RFC] non-preemptible kernel socket for RAMster

[Loke, Chetan]

> -----Original Message-----
> From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
> Sent: July 05, 2011 3:19 PM
> To: Loke, Chetan; netdev@vger.kernel.org
> Cc: Konrad Wilk; linux-mm
> Subject: RE: [RFC] non-preemptible kernel socket for RAMster
> 

> Actually, RAMster is using a much more flexible type of
> RAM-drive; it is built on top of Transcendent Memory
> and on top of zcache (and thus on top of cleancache and
> frontswap).  A RAM-drive is fixed size so is not very suitable
> for the flexibility required for RAMster.  For example,
> suppose you have two machines A and B.  At one point in
> time A is overcommitted and needs to swap and B is relatively
> idle.  Then later, B is overcommitted and needs to swap and
> A is relatively idle.  RAMster can handle this entirely
> dynamically, a RAM-drive cannot.


Again, iff NBD works with a ram-drive then you really wouldn't need to
do anything. How often are you going to re-size your remote-SWAP?  Plus,
you can make nbd-server listen on multiple ports - Google(Linux NBD)
returned: http://www.fi.muni.cz/~kripac/orac-nbd/ . Look at the
nbd-server code to see if it launches multiple kernel-threads for
servicing different ports. If not, one can enhance it and scale that way
too. But nbd-server today can service multiple-ports(that is effectively
servicing multiple clients). So why not add NBD-filesystem-filters to
make it point to local/remote swap?


> 
> Thanks.  Could you provide a pointer for this?  I found
> the SCST sourceforge page but no obvious references to
> scst-in-ram-mode.  (But also, since it appears to be
> SCSI-related, I wonder if it also assumes a fixed size
> target device, RAM or disk or ??)
> 

Yes, it is SCSI. You should be looking for SCST I/O modes. Read some
docs and then send an email to the scst-mailing-list. If you speak about
block-IO-performance then FC(in its class of price/performance factor)
is more than capable of handling any workload. FC is a protocol designed
for storage. No exotic fabric other than FC is needed.
Folks who start with ethernet for block-IO, always start with bare
minimal code and then for squeezing block-IO performance(aka version 2
of the product), keep hacking repeatedly or go for a link-speed upgrade.
Start with FC, period.


> Dan

Chetan Loke

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

RE: [RFC] non-preemptible kernel socket for RAMster

[Dan Magenheimer]

> From: Loke, Chetan [mailto:Chetan.Loke@netscout.com]
> Subject: RE: [RFC] non-preemptible kernel socket for RAMster
> 
> > From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
> 
> > Actually, RAMster is using a much more flexible type of
> > RAM-drive; it is built on top of Transcendent Memory
> > and on top of zcache (and thus on top of cleancache and
> > frontswap).  A RAM-drive is fixed size so is not very suitable
> > for the flexibility required for RAMster.  For example,
> > suppose you have two machines A and B.  At one point in
> > time A is overcommitted and needs to swap and B is relatively
> > idle.  Then later, B is overcommitted and needs to swap and
> > A is relatively idle.  RAMster can handle this entirely
> > dynamically, a RAM-drive cannot.
> 
> Again, iff NBD works with a ram-drive then you really wouldn't need to
> do anything. How often are you going to re-size your remote-SWAP?  Plus,
> you can make nbd-server listen on multiple ports - Google(Linux NBD)
> returned: http://www.fi.muni.cz/~kripac/orac-nbd/ . Look at the
> nbd-server code to see if it launches multiple kernel-threads for
> servicing different ports. If not, one can enhance it and scale that way
> too. But nbd-server today can service multiple-ports(that is effectively
> servicing multiple clients). So why not add NBD-filesystem-filters to
> make it point to local/remote swap?

Well, we may be talking past each other, but the RAMster answer to:

> How often are you going to re-size your remote-SWAP?

is "as often as the working set changes on any machine in the
cluster", meaning *constantly*, entirely dynamically!  How
about a more specific example:  Suppose you have 2 machines,
each with 8GB of memory.  99% of the time each machine is
chugging along just fine and doesn't really need more than 4GB,
and may even use less than 1GB a large part of the time.
But very now and then, one of the machines randomly needs
9GB, 10GB, maybe even 12GB  of memory.  This would normally
result in swapping.  (Most system administrators won't even
have this much information... they'll just know they are
seeing swapping and decide they need to buy more RAM.)

With NBD to a ram-drive, each machine would need to pre-allocate
4GB of RAM for the RAM-drive, leaving only 4GB of RAM for
the "local" RAM.  The result will actually be MORE swapping
because a fixed amount of RAM has been pre-reserved for the
other machine's swap.   With RAMster, everything is done dynamically,
so all that matters is the maximum of the sum of the RAM
used.  You may even be able to *remove* ~2GB of RAM from each
of the systems and still never see any swapping to disk.

> > Thanks.  Could you provide a pointer for this?  I found
> > the SCST sourceforge page but no obvious references to
> > scst-in-ram-mode.  (But also, since it appears to be
> > SCSI-related, I wonder if it also assumes a fixed size
> > target device, RAM or disk or ??)
> 
> Yes, it is SCSI. You should be looking for SCST I/O modes. Read some
> docs and then send an email to the scst-mailing-list. If you speak about
> block-IO-performance then FC(in its class of price/performance factor)
> is more than capable of handling any workload. FC is a protocol designed
> for storage. No exotic fabric other than FC is needed.
> Folks who start with ethernet for block-IO, always start with bare
> minimal code and then for squeezing block-IO performance(aka version 2
> of the product), keep hacking repeatedly or go for a link-speed upgrade.
> Start with FC, period.

My point was that block I/O devices (AFAIK) always present a fixed
"size" to the kernel, and if this is also true of scst-in-ram-mode,
the same problem as swap-over-NBD occurs... it's not dynamic.
RAMster does not present a block-I/O storage-like interface;
it's using the Transcendent Memory interface, which is designed
for "slow RAM" of an unknown-and-dynamic size.

I'm not a storage expert either, but I do wonder if "no exotic
fabric other than FC" isn't an oxymoron ;-)  FC is certainly
too exotic for me.

Dan

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

RE: [RFC] non-preemptible kernel socket for RAMster

[Loke, Chetan]

> -----Original Message-----
> From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
> Sent: July 05, 2011 9:06 PM
> To: Loke, Chetan; netdev@vger.kernel.org
> Cc: Konrad Wilk; linux-mm
> Subject: RE: [RFC] non-preemptible kernel socket for RAMster
> 
> > From: Loke, Chetan [mailto:Chetan.Loke@netscout.com]
> > Subject: RE: [RFC] non-preemptible kernel socket for RAMster
> >
> > > From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
> 
> > How often are you going to re-size your remote-SWAP?
> 
> is "as often as the working set changes on any machine in the
> cluster", meaning *constantly*, entirely dynamically!  How
> about a more specific example:  Suppose you have 2 machines,
> each with 8GB of memory.  99% of the time each machine is
> chugging along just fine and doesn't really need more than 4GB,
> and may even use less than 1GB a large part of the time.
> But very now and then, one of the machines randomly needs
> 9GB, 10GB, maybe even 12GB  of memory.  This would normally
> result in swapping.  (Most system administrators won't even
> have this much information... they'll just know they are
> seeing swapping and decide they need to buy more RAM.)
> 

Ok, I understand there is interest in implementing
'remote-volatile-ballooning-variant' but how do you pick a remote
candidate(hypervisor)? Let's say, memory could be available on remote
system but what if the remote-p{NIC,CPU} is overloaded? Sure, sysadmins
won't have this info because this so dynamic(and it's quite possible as
you mentioned above). But does the trans-remote-API know about this
resource-availability before opening a remote-channel?

Stressing the remote-p{NIC/CPU} might trick hypervisor-vmotion-plugin to
vmotion VM[s] to another hypervisor. How is trans-remote-API integrating
with remote/global vmotion policies to avoid this false vmotion?


> Dan

Chetan Loke

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

RE: [RFC] non-preemptible kernel socket for RAMster

[Dan Magenheimer]

> From: Loke, Chetan [mailto:Chetan.Loke@netscout.com]
> Subject: RE: [RFC] non-preemptible kernel socket for RAMster
> 
> > -----Original Message-----
> > From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
> >
> > > From: Loke, Chetan [mailto:Chetan.Loke@netscout.com]
> > >
> > > > From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
> >
> > > How often are you going to re-size your remote-SWAP?
> >
> > is "as often as the working set changes on any machine in the
> > cluster", meaning *constantly*, entirely dynamically!  How
> > about a more specific example:  Suppose you have 2 machines,
> > each with 8GB of memory.  99% of the time each machine is
> > chugging along just fine and doesn't really need more than 4GB,
> > and may even use less than 1GB a large part of the time.
> > But very now and then, one of the machines randomly needs
> > 9GB, 10GB, maybe even 12GB  of memory.  This would normally
> > result in swapping.  (Most system administrators won't even
> > have this much information... they'll just know they are
> > seeing swapping and decide they need to buy more RAM.)
> >
> 
> Ok, I understand there is interest in implementing
> 'remote-volatile-ballooning-variant' but how do you pick a remote
> candidate(hypervisor)? Let's say, memory could be available on remote
> system but what if the remote-p{NIC,CPU} is overloaded? Sure, sysadmins
> won't have this info because this so dynamic(and it's quite possible as
> you mentioned above). But does the trans-remote-API know about this
> resource-availability before opening a remote-channel?
> 
> Stressing the remote-p{NIC/CPU} might trick hypervisor-vmotion-plugin to
> vmotion VM[s] to another hypervisor. How is trans-remote-API integrating
> with remote/global vmotion policies to avoid this false vmotion?

Hi Chetan --

Thanks for the continued discussion.

First, let me clarify that RAMster does not depend on virtualization.
At some time in the future, it may be a nice addition for KVM*,
but the version I am developing currently only works on a
cluster of physical machines.  So vmotion/migration is not
an issue right now


As for choosing the remote machine, another key feature of
the Transcendent Memory mechanism is that any and every page
can be rejected.  If rejected, the page remains local.  In
essence, on *every* page-to-be-swapped, machine A *asks*
machine B, "can you take this page"?  If the answer is no,
machine A can choose another machine (C), or may choose to
swap the page to its own slow swap disk.  (Currently,
only the latter is implemented, but more complicated
policy could certainly be implemented.)

Dan

* Xen doesn't have drivers so RAMster-over-network is not an option
for Xen.  A future RAMster-over-exofabric might work with Xen though.)
And, by the way, the Transcendent Memory implementation on Xen
does handle vmotion/migration so it is a solvable problem.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>