From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail6.bemta7.messagelabs.com (mail6.bemta7.messagelabs.com [216.82.255.55]) by kanga.kvack.org (Postfix) with ESMTP id 56DAF900134 for ; Tue, 5 Jul 2011 11:54:33 -0400 (EDT) MIME-Version: 1.0 Message-ID: <4232c4b6-15be-42d8-be42-6e27f9188ce2@default> Date: Tue, 5 Jul 2011 08:54:06 -0700 (PDT) From: Dan Magenheimer Subject: [RFC] non-preemptible kernel socket for RAMster Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Sender: owner-linux-mm@kvack.org List-ID: To: netdev@vger.kernel.org Cc: Konrad Wilk , linux-mm In working on a kernel project called RAMster* (where RAM on a remote system may be used for clean page cache pages and for swap pages), I found I have need for a kernel socket to be used when in non-preemptible state. I admit to being a networking idiot, but I have been successfully using the following small patch. I'm not sure whether I am lucky so far... perhaps more sockets or larger/different loads will require a lot more changes (or maybe even make my objective impossible). So I thought I'd post it for comment. I'd appreciate any thoughts or suggestions. Thanks, Dan * http://events.linuxfoundation.org/events/linuxcon/magenheimer=20 diff -Napur linux-2.6.37/net/core/sock.c linux-2.6.37-ramster/net/core/sock= .c --- linux-2.6.37/net/core/sock.c=092011-07-03 19:14:52.267853088 -0600 +++ linux-2.6.37-ramster/net/core/sock.c=092011-07-03 19:10:04.340980799 -0= 600 @@ -1587,6 +1587,14 @@ static void __lock_sock(struct sock *sk) =09__acquires(&sk->sk_lock.slock) { =09DEFINE_WAIT(wait); +=09if (!preemptible()) { +=09=09while (sock_owned_by_user(sk)) { +=09=09=09spin_unlock_bh(&sk->sk_lock.slock); +=09=09=09cpu_relax(); +=09=09=09spin_lock_bh(&sk->sk_lock.slock); +=09=09} +=09=09return; +=09} =20 =09for (;;) { =09=09prepare_to_wait_exclusive(&sk->sk_lock.wq, &wait, @@ -1623,7 +1631,8 @@ static void __release_sock(struct sock * =09=09=09 * This is safe to do because we've taken the backlog =09=09=09 * queue private: =09=09=09 */ -=09=09=09cond_resched_softirq(); +=09=09=09if (preemptible()) +=09=09=09=09cond_resched_softirq(); =09=09=09skb =3D next; =09=09} while (skb !=3D NULL); -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail138.messagelabs.com (mail138.messagelabs.com [216.82.249.35]) by kanga.kvack.org (Postfix) with ESMTP id 7BFF1900134 for ; Tue, 5 Jul 2011 12:30:47 -0400 (EDT) Received: by wyg36 with SMTP id 36so5506901wyg.14 for ; Tue, 05 Jul 2011 09:30:44 -0700 (PDT) Subject: Re: [RFC] non-preemptible kernel socket for RAMster From: Eric Dumazet In-Reply-To: <4232c4b6-15be-42d8-be42-6e27f9188ce2@default> References: <4232c4b6-15be-42d8-be42-6e27f9188ce2@default> Content-Type: text/plain; charset="UTF-8" Date: Tue, 05 Jul 2011 18:30:30 +0200 Message-ID: <1309883430.2271.27.camel@edumazet-HP-Compaq-6005-Pro-SFF-PC> Mime-Version: 1.0 Content-Transfer-Encoding: 8bit Sender: owner-linux-mm@kvack.org List-ID: To: Dan Magenheimer Cc: netdev@vger.kernel.org, Konrad Wilk , linux-mm Le mardi 05 juillet 2011 A 08:54 -0700, Dan Magenheimer a A(C)crit : > In working on a kernel project called RAMster* (where RAM on a > remote system may be used for clean page cache pages and for swap > pages), I found I have need for a kernel socket to be used when > in non-preemptible state. I admit to being a networking idiot, > but I have been successfully using the following small patch. > I'm not sure whether I am lucky so far... perhaps more > sockets or larger/different loads will require a lot more > changes (or maybe even make my objective impossible). > So I thought I'd post it for comment. I'd appreciate > any thoughts or suggestions. > > Thanks, > Dan > > * http://events.linuxfoundation.org/events/linuxcon/magenheimer > > diff -Napur linux-2.6.37/net/core/sock.c linux-2.6.37-ramster/net/core/sock.c > --- linux-2.6.37/net/core/sock.c 2011-07-03 19:14:52.267853088 -0600 > +++ linux-2.6.37-ramster/net/core/sock.c 2011-07-03 19:10:04.340980799 -0600 > @@ -1587,6 +1587,14 @@ static void __lock_sock(struct sock *sk) > __acquires(&sk->sk_lock.slock) > { > DEFINE_WAIT(wait); > + if (!preemptible()) { > + while (sock_owned_by_user(sk)) { > + spin_unlock_bh(&sk->sk_lock.slock); > + cpu_relax(); > + spin_lock_bh(&sk->sk_lock.slock); > + } > + return; > + } Hmm, was this tested on UP machine ? > > for (;;) { > prepare_to_wait_exclusive(&sk->sk_lock.wq, &wait, > @@ -1623,7 +1631,8 @@ static void __release_sock(struct sock * > * This is safe to do because we've taken the backlog > * queue private: > */ > - cond_resched_softirq(); > + if (preemptible()) > + cond_resched_softirq(); > skb = next; > } while (skb != NULL); -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail6.bemta8.messagelabs.com (mail6.bemta8.messagelabs.com [216.82.243.55]) by kanga.kvack.org (Postfix) with ESMTP id 5FEC6900134 for ; Tue, 5 Jul 2011 12:36:54 -0400 (EDT) Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Subject: RE: [RFC] non-preemptible kernel socket for RAMster Date: Tue, 5 Jul 2011 12:36:43 -0400 Message-ID: In-Reply-To: <4232c4b6-15be-42d8-be42-6e27f9188ce2@default> References: <4232c4b6-15be-42d8-be42-6e27f9188ce2@default> From: "Loke, Chetan" Sender: owner-linux-mm@kvack.org List-ID: To: Dan Magenheimer , netdev@vger.kernel.org Cc: Konrad Wilk , linux-mm > -----Original Message----- > From: netdev-owner@vger.kernel.org [mailto:netdev- > owner@vger.kernel.org] On Behalf Of Dan Magenheimer > Sent: July 05, 2011 11:54 AM > To: netdev@vger.kernel.org > Cc: Konrad Wilk; linux-mm > Subject: [RFC] non-preemptible kernel socket for RAMster >=20 > In working on a kernel project called RAMster* (where RAM on a > remote system may be used for clean page cache pages and for swap > pages), I found I have need for a kernel socket to be used when How is RAMster+swap different than NBD's (pending etc?)support for SWAP over NBD? Chetan Loke -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail138.messagelabs.com (mail138.messagelabs.com [216.82.249.35]) by kanga.kvack.org (Postfix) with ESMTP id 176386B004A for ; Tue, 5 Jul 2011 13:25:35 -0400 (EDT) MIME-Version: 1.0 Message-ID: <918f7b76-4904-41cc-9f55-c07adafb34b4@default> Date: Tue, 5 Jul 2011 10:25:09 -0700 (PDT) From: Dan Magenheimer Subject: RE: [RFC] non-preemptible kernel socket for RAMster References: <4232c4b6-15be-42d8-be42-6e27f9188ce2@default 1309883430.2271.27.camel@edumazet-HP-Compaq-6005-Pro-SFF-PC> In-Reply-To: <1309883430.2271.27.camel@edumazet-HP-Compaq-6005-Pro-SFF-PC> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Sender: owner-linux-mm@kvack.org List-ID: To: Eric Dumazet Cc: netdev@vger.kernel.org, Konrad Wilk , linux-mm > From: Eric Dumazet [mailto:eric.dumazet@gmail.com] > Sent: Tuesday, July 05, 2011 10:31 AM > To: Dan Magenheimer > Cc: netdev@vger.kernel.org; Konrad Wilk; linux-mm > Subject: Re: [RFC] non-preemptible kernel socket for RAMster >=20 > Le mardi 05 juillet 2011 =C3=A0 08:54 -0700, Dan Magenheimer a =C3=A9crit= : > > In working on a kernel project called RAMster* (where RAM on a > > remote system may be used for clean page cache pages and for swap > > pages), I found I have need for a kernel socket to be used when > > in non-preemptible state. I admit to being a networking idiot, > > but I have been successfully using the following small patch. > > I'm not sure whether I am lucky so far... perhaps more > > sockets or larger/different loads will require a lot more > > changes (or maybe even make my objective impossible). > > So I thought I'd post it for comment. I'd appreciate > > any thoughts or suggestions. > > > > Thanks, > > Dan > > > > * http://events.linuxfoundation.org/events/linuxcon/magenheimer > > > > diff -Napur linux-2.6.37/net/core/sock.c linux-2.6.37-ramster/net/core/= sock.c > > --- linux-2.6.37/net/core/sock.c=092011-07-03 19:14:52.267853088 -0600 > > +++ linux-2.6.37-ramster/net/core/sock.c=092011-07-03 19:10:04.34098079= 9 -0600 > > @@ -1587,6 +1587,14 @@ static void __lock_sock(struct sock *sk) > > =09__acquires(&sk->sk_lock.slock) > > { > > =09DEFINE_WAIT(wait); > > +=09if (!preemptible()) { > > +=09=09while (sock_owned_by_user(sk)) { > > +=09=09=09spin_unlock_bh(&sk->sk_lock.slock); > > +=09=09=09cpu_relax(); > > +=09=09=09spin_lock_bh(&sk->sk_lock.slock); > > +=09=09} > > +=09=09return; > > +=09} >=20 > Hmm, was this tested on UP machine ? Hi Eric -- Thanks for the reply! I hadn't tested UP in awhile so am testing now, and it seems to work OK so far. However, I am just testing my socket, *not* testing sockets in general. Are you implying that this patch will break (kernel) sockets in general on a UP machine? If so, could you be more specific as to why? (Again, I said I am a networking idiot. ;-) I played a bit with adding a new SOCK_ flag and triggering off of that, but this version of the patch seemed much simpler. Thanks, Dan -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail6.bemta8.messagelabs.com (mail6.bemta8.messagelabs.com [216.82.243.55]) by kanga.kvack.org (Postfix) with ESMTP id C1FE56B007E for ; Tue, 5 Jul 2011 13:25:45 -0400 (EDT) MIME-Version: 1.0 Message-ID: <6147447c-ecab-43ea-9b4a-1ff64b2089f0@default> Date: Tue, 5 Jul 2011 10:25:21 -0700 (PDT) From: Dan Magenheimer Subject: RE: [RFC] non-preemptible kernel socket for RAMster References: <4232c4b6-15be-42d8-be42-6e27f9188ce2@default D3F292ADF945FB49B35E96C94C2061B91257D65C@nsmail.netscout.com> In-Reply-To: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Sender: owner-linux-mm@kvack.org List-ID: To: "Loke, Chetan" , netdev@vger.kernel.org Cc: Konrad Wilk , linux-mm > From: Loke, Chetan [mailto:Chetan.Loke@netscout.com] > Sent: Tuesday, July 05, 2011 10:37 AM > To: Dan Magenheimer; netdev@vger.kernel.org > Cc: Konrad Wilk; linux-mm > Subject: RE: [RFC] non-preemptible kernel socket for RAMster >=20 > > In working on a kernel project called RAMster* (where RAM on a > > remote system may be used for clean page cache pages and for swap > > pages), I found I have need for a kernel socket to be used when >=20 > How is RAMster+swap different than NBD's (pending etc?)support for SWAP > over NBD? Hi Chetan -- Thanks for your question. I may be ignorant of details about NBD, but did some quick research using google. If I understand correctly, swap over NBD is still writing to a configured swap disk on the remote machine. RAMster is swapping to *RAM* on the remote machine. The idea is that most machines are very overprovisioned in RAM, and are rarely using all of their RAM, especially when a machine is (mostly) idle. In other words, the "max of the sums" of RAM usage on a group of machines is much lower than the "sum of the max" of RAM usage. So if the network is sufficiently faster than disk for moving a page of data, RAMster provides a significant performance improvement. OR RAMster may allow a significant reduction in the total amount of RAM across a data center. The version of RAMster I am working on now is really a proof-of-concept that works over sockets, using the ocfs2 cluster layer. One can easily envision a future "exo-fabric" which allows one machine to write to the RAM of another machine... for this future hardware, RAMster becomes much more interesting. Thanks, Dan -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail138.messagelabs.com (mail138.messagelabs.com [216.82.249.35]) by kanga.kvack.org (Postfix) with SMTP id 970C46B004A for ; Tue, 5 Jul 2011 13:52:29 -0400 (EDT) Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Subject: RE: [RFC] non-preemptible kernel socket for RAMster Date: Tue, 5 Jul 2011 13:52:17 -0400 Message-ID: In-Reply-To: <6147447c-ecab-43ea-9b4a-1ff64b2089f0@default> References: <4232c4b6-15be-42d8-be42-6e27f9188ce2@default D3F292ADF945FB49B35E96C94C2061B91257D65C@nsmail.netscout.com> <6147447c-ecab-43ea-9b4a-1ff64b2089f0@default> From: "Loke, Chetan" Sender: owner-linux-mm@kvack.org List-ID: To: Dan Magenheimer , netdev@vger.kernel.org Cc: Konrad Wilk , linux-mm > -----Original Message----- > From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] > Sent: July 05, 2011 1:25 PM > To: Loke, Chetan; netdev@vger.kernel.org > Cc: Konrad Wilk; linux-mm > Subject: RE: [RFC] non-preemptible kernel socket for RAMster >=20 > > From: Loke, Chetan [mailto:Chetan.Loke@netscout.com] > > Sent: Tuesday, July 05, 2011 10:37 AM > > To: Dan Magenheimer; netdev@vger.kernel.org > > Cc: Konrad Wilk; linux-mm > > Subject: RE: [RFC] non-preemptible kernel socket for RAMster > > > > > In working on a kernel project called RAMster* (where RAM on a > > > remote system may be used for clean page cache pages and for swap > > > pages), I found I have need for a kernel socket to be used when > > > > How is RAMster+swap different than NBD's (pending etc?)support for > SWAP > > over NBD? >=20 > Hi Chetan -- >=20 > Thanks for your question. >=20 > I may be ignorant of details about NBD, but did some quick > research using google. If I understand correctly, swap over > NBD is still writing to a configured swap disk on the remote Hi - I thought NBD-server needs a backing store(a file).=20 Now the file itself could reside on a RAM-drive or disk-drive etc. And so a remote NBD(disk or RAM) can be mounted locally as a swap device. The local client should still see it as a block device. I haven't used the RAM-drive feature myself but you may want to check if it works or even borrow that logic in your code. > machine. RAMster is swapping to *RAM* on the remote machine. > The idea is that most machines are very overprovisioned in > RAM, and are rarely using all of their RAM, especially when > a machine is (mostly) idle. In other words, the "max of > the sums" of RAM usage on a group of machines is much lower > than the "sum of the max" of RAM usage. >=20 > So if the network is sufficiently faster than disk for > moving a page of data, RAMster provides a significant > performance improvement. OR RAMster may allow a significant > reduction in the total amount of RAM across a data center. >=20 > The version of RAMster I am working on now is really > a proof-of-concept that works over sockets, using the > ocfs2 cluster layer. One can easily envision a future > "exo-fabric" which allows one machine to write to the > RAM of another machine... for this future hardware, > RAMster becomes much more interesting. >=20 Or you can also try scst-in-RAM mode(if you want to experiment with different fabrics). > Thanks, > Dan Thanks Chetan Loke -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail143.messagelabs.com (mail143.messagelabs.com [216.82.254.35]) by kanga.kvack.org (Postfix) with ESMTP id 37F7B9000C2 for ; Tue, 5 Jul 2011 14:24:08 -0400 (EDT) Received: by wyg36 with SMTP id 36so5598989wyg.14 for ; Tue, 05 Jul 2011 11:24:02 -0700 (PDT) Subject: RE: [RFC] non-preemptible kernel socket for RAMster From: Eric Dumazet In-Reply-To: <918f7b76-4904-41cc-9f55-c07adafb34b4@default> References: <4232c4b6-15be-42d8-be42-6e27f9188ce2@default 1309883430.2271.27.camel@edumazet-HP-Compaq-6005-Pro-SFF-PC> <918f7b76-4904-41cc-9f55-c07adafb34b4@default> Content-Type: text/plain; charset="UTF-8" Date: Tue, 05 Jul 2011 20:23:59 +0200 Message-ID: <1309890239.2545.10.camel@edumazet-laptop> Mime-Version: 1.0 Content-Transfer-Encoding: 8bit Sender: owner-linux-mm@kvack.org List-ID: To: Dan Magenheimer Cc: netdev@vger.kernel.org, Konrad Wilk , linux-mm Le mardi 05 juillet 2011 A 10:25 -0700, Dan Magenheimer a A(C)crit : > > From: Eric Dumazet [mailto:eric.dumazet@gmail.com] > > Sent: Tuesday, July 05, 2011 10:31 AM > > To: Dan Magenheimer > > Cc: netdev@vger.kernel.org; Konrad Wilk; linux-mm > > Subject: Re: [RFC] non-preemptible kernel socket for RAMster > > > > Le mardi 05 juillet 2011 A 08:54 -0700, Dan Magenheimer a A(C)crit : > > > In working on a kernel project called RAMster* (where RAM on a > > > remote system may be used for clean page cache pages and for swap > > > pages), I found I have need for a kernel socket to be used when > > > in non-preemptible state. I admit to being a networking idiot, > > > but I have been successfully using the following small patch. > > > I'm not sure whether I am lucky so far... perhaps more > > > sockets or larger/different loads will require a lot more > > > changes (or maybe even make my objective impossible). > > > So I thought I'd post it for comment. I'd appreciate > > > any thoughts or suggestions. > > > > > > Thanks, > > > Dan > > > > > > * http://events.linuxfoundation.org/events/linuxcon/magenheimer > > > > > > diff -Napur linux-2.6.37/net/core/sock.c linux-2.6.37-ramster/net/core/sock.c > > > --- linux-2.6.37/net/core/sock.c 2011-07-03 19:14:52.267853088 -0600 > > > +++ linux-2.6.37-ramster/net/core/sock.c 2011-07-03 19:10:04.340980799 -0600 > > > @@ -1587,6 +1587,14 @@ static void __lock_sock(struct sock *sk) > > > __acquires(&sk->sk_lock.slock) > > > { > > > DEFINE_WAIT(wait); > > > + if (!preemptible()) { > > > + while (sock_owned_by_user(sk)) { > > > + spin_unlock_bh(&sk->sk_lock.slock); > > > + cpu_relax(); > > > + spin_lock_bh(&sk->sk_lock.slock); > > > + } > > > + return; > > > + } > > > > Hmm, was this tested on UP machine ? > > Hi Eric -- > > Thanks for the reply! > > I hadn't tested UP in awhile so am testing now, and it seems to > work OK so far. However, I am just testing my socket, *not* testing > sockets in general. Are you implying that this patch will > break (kernel) sockets in general on a UP machine? If so, > could you be more specific as to why? (Again, I said > I am a networking idiot. ;-) I played a bit with adding > a new SOCK_ flag and triggering off of that, but this > version of the patch seemed much simpler. Say you have two processes and socket S One process locks socket S, and is preempted by another process. This second process is non preemptible and try to lock same socket. -> deadlock, since P1 never releases socket S -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail203.messagelabs.com (mail203.messagelabs.com [216.82.254.243]) by kanga.kvack.org (Postfix) with ESMTP id E2EA59000C2 for ; Tue, 5 Jul 2011 15:07:36 -0400 (EDT) MIME-Version: 1.0 Message-ID: Date: Tue, 5 Jul 2011 12:07:10 -0700 (PDT) From: Dan Magenheimer Subject: RE: [RFC] non-preemptible kernel socket for RAMster References: <4232c4b6-15be-42d8-be42-6e27f9188ce2@default> <1309883430.2271.27.camel@edumazet-HP-Compaq-6005-Pro-SFF-PC> <918f7b76-4904-41cc-9f55-c07adafb34b4@default 1309890239.2545.10.camel@edumazet-laptop> In-Reply-To: <1309890239.2545.10.camel@edumazet-laptop> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Sender: owner-linux-mm@kvack.org List-ID: To: Eric Dumazet Cc: netdev@vger.kernel.org, Konrad Wilk , linux-mm > > > > +++ linux-2.6.37-ramster/net/core/sock.c=092011-07-03 19:10:04.3409= 80799 -0600 > > > > @@ -1587,6 +1587,14 @@ static void __lock_sock(struct sock *sk) > > > > =09__acquires(&sk->sk_lock.slock) > > > > { > > > > =09DEFINE_WAIT(wait); > > > > +=09if (!preemptible()) { > > > > +=09=09while (sock_owned_by_user(sk)) { > > > > +=09=09=09spin_unlock_bh(&sk->sk_lock.slock); > > > > +=09=09=09cpu_relax(); > > > > +=09=09=09spin_lock_bh(&sk->sk_lock.slock); > > > > +=09=09} > > > > +=09=09return; > > > > +=09} > > > > > > Hmm, was this tested on UP machine ? > > > > Hi Eric -- > > > > Thanks for the reply! > > > > I hadn't tested UP in awhile so am testing now, and it seems to > > work OK so far. However, I am just testing my socket, *not* testing > > sockets in general. Are you implying that this patch will > > break (kernel) sockets in general on a UP machine? If so, > > could you be more specific as to why? (Again, I said > > I am a networking idiot. ;-) I played a bit with adding > > a new SOCK_ flag and triggering off of that, but this > > version of the patch seemed much simpler. >=20 > Say you have two processes and socket S >=20 > One process locks socket S, and is preempted by another process. >=20 > This second process is non preemptible and try to lock same socket. >=20 > -> deadlock, since P1 never releases socket S Oh, OK. My use model is that a socket that is used non-preemptible must always be used non-preemptible. In other words, this kind of socket is an extreme form of non-blocking. Doesn't that seem like a reasonable constraint?=20 Thanks, Dan -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail6.bemta8.messagelabs.com (mail6.bemta8.messagelabs.com [216.82.243.55]) by kanga.kvack.org (Postfix) with ESMTP id B76579000C2 for ; Tue, 5 Jul 2011 15:19:00 -0400 (EDT) MIME-Version: 1.0 Message-ID: <704d094e-7b81-480f-8363-327218d1b0ea@default> Date: Tue, 5 Jul 2011 12:18:36 -0700 (PDT) From: Dan Magenheimer Subject: RE: [RFC] non-preemptible kernel socket for RAMster References: <4232c4b6-15be-42d8-be42-6e27f9188ce2@default> <6147447c-ecab-43ea-9b4a-1ff64b2089f0@default D3F292ADF945FB49B35E96C94C2061B91257D6FD@nsmail.netscout.com> In-Reply-To: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Sender: owner-linux-mm@kvack.org List-ID: To: "Loke, Chetan" , netdev@vger.kernel.org Cc: Konrad Wilk , linux-mm > From: Loke, Chetan [mailto:Chetan.Loke@netscout.com] > > From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] > > Subject: RE: [RFC] non-preemptible kernel socket for RAMster > > > > > From: Loke, Chetan [mailto:Chetan.Loke@netscout.com] > > > Sent: Tuesday, July 05, 2011 10:37 AM > > > To: Dan Magenheimer; netdev@vger.kernel.org > > > Cc: Konrad Wilk; linux-mm > > > Subject: RE: [RFC] non-preemptible kernel socket for RAMster > > > > > > > In working on a kernel project called RAMster* (where RAM on a > > > > remote system may be used for clean page cache pages and for swap > > > > pages), I found I have need for a kernel socket to be used when > > > > > > How is RAMster+swap different than NBD's (pending etc?)support for > > > SWAP over NBD? > > > > I may be ignorant of details about NBD, but did some quick > > research using google. If I understand correctly, swap over > > NBD is still writing to a configured swap disk on the remote >=20 > Hi - I thought NBD-server needs a backing store(a file). > Now the file itself could reside on a RAM-drive or disk-drive etc. > And so a remote NBD(disk or RAM) can be mounted locally as a swap > device. > The local client should still see it as a block device. >=20 > I haven't used the RAM-drive feature myself but you may want to check if > it > works or even borrow that logic in your code. Actually, RAMster is using a much more flexible type of RAM-drive; it is built on top of Transcendent Memory and on top of zcache (and thus on top of cleancache and frontswap). A RAM-drive is fixed size so is not very suitable for the flexibility required for RAMster. For example, suppose you have two machines A and B. At one point in time A is overcommitted and needs to swap and B is relatively idle. Then later, B is overcommitted and needs to swap and A is relatively idle. RAMster can handle this entirely dynamically, a RAM-drive cannot. > > machine. RAMster is swapping to *RAM* on the remote machine. > > The idea is that most machines are very overprovisioned in > > RAM, and are rarely using all of their RAM, especially when > > a machine is (mostly) idle. In other words, the "max of > > the sums" of RAM usage on a group of machines is much lower > > than the "sum of the max" of RAM usage. > > > > So if the network is sufficiently faster than disk for > > moving a page of data, RAMster provides a significant > > performance improvement. OR RAMster may allow a significant > > reduction in the total amount of RAM across a data center. > > > > The version of RAMster I am working on now is really > > a proof-of-concept that works over sockets, using the > > ocfs2 cluster layer. One can easily envision a future > > "exo-fabric" which allows one machine to write to the > > RAM of another machine... for this future hardware, > > RAMster becomes much more interesting. >=20 > Or you can also try scst-in-RAM mode(if you want to experiment with > different fabrics). Thanks. Could you provide a pointer for this? I found the SCST sourceforge page but no obvious references to scst-in-ram-mode. (But also, since it appears to be SCSI-related, I wonder if it also assumes a fixed size target device, RAM or disk or ??) Dan -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail143.messagelabs.com (mail143.messagelabs.com [216.82.254.35]) by kanga.kvack.org (Postfix) with SMTP id 69A9E9000C2 for ; Tue, 5 Jul 2011 18:27:27 -0400 (EDT) Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Subject: RE: [RFC] non-preemptible kernel socket for RAMster Date: Tue, 5 Jul 2011 18:27:22 -0400 Message-ID: In-Reply-To: <704d094e-7b81-480f-8363-327218d1b0ea@default> References: <4232c4b6-15be-42d8-be42-6e27f9188ce2@default> <6147447c-ecab-43ea-9b4a-1ff64b2089f0@default D3F292ADF945FB49B35E96C94C2061B91257D6FD@nsmail.netscout.com> <704d094e-7b81-480f-8363-327218d1b0ea@default> From: "Loke, Chetan" Sender: owner-linux-mm@kvack.org List-ID: To: Dan Magenheimer , netdev@vger.kernel.org Cc: Konrad Wilk , linux-mm > -----Original Message----- > From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] > Sent: July 05, 2011 3:19 PM > To: Loke, Chetan; netdev@vger.kernel.org > Cc: Konrad Wilk; linux-mm > Subject: RE: [RFC] non-preemptible kernel socket for RAMster >=20 > Actually, RAMster is using a much more flexible type of > RAM-drive; it is built on top of Transcendent Memory > and on top of zcache (and thus on top of cleancache and > frontswap). A RAM-drive is fixed size so is not very suitable > for the flexibility required for RAMster. For example, > suppose you have two machines A and B. At one point in > time A is overcommitted and needs to swap and B is relatively > idle. Then later, B is overcommitted and needs to swap and > A is relatively idle. RAMster can handle this entirely > dynamically, a RAM-drive cannot. Again, iff NBD works with a ram-drive then you really wouldn't need to do anything. How often are you going to re-size your remote-SWAP? Plus, you can make nbd-server listen on multiple ports - Google(Linux NBD) returned: http://www.fi.muni.cz/~kripac/orac-nbd/ . Look at the nbd-server code to see if it launches multiple kernel-threads for servicing different ports. If not, one can enhance it and scale that way too. But nbd-server today can service multiple-ports(that is effectively servicing multiple clients). So why not add NBD-filesystem-filters to make it point to local/remote swap? >=20 > Thanks. Could you provide a pointer for this? I found > the SCST sourceforge page but no obvious references to > scst-in-ram-mode. (But also, since it appears to be > SCSI-related, I wonder if it also assumes a fixed size > target device, RAM or disk or ??) >=20 Yes, it is SCSI. You should be looking for SCST I/O modes. Read some docs and then send an email to the scst-mailing-list. If you speak about block-IO-performance then FC(in its class of price/performance factor) is more than capable of handling any workload. FC is a protocol designed for storage. No exotic fabric other than FC is needed. Folks who start with ethernet for block-IO, always start with bare minimal code and then for squeezing block-IO performance(aka version 2 of the product), keep hacking repeatedly or go for a link-speed upgrade. Start with FC, period. > Dan Chetan Loke -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail143.messagelabs.com (mail143.messagelabs.com [216.82.254.35]) by kanga.kvack.org (Postfix) with ESMTP id E475D9000C2 for ; Tue, 5 Jul 2011 21:06:19 -0400 (EDT) MIME-Version: 1.0 Message-ID: Date: Tue, 5 Jul 2011 18:05:53 -0700 (PDT) From: Dan Magenheimer Subject: RE: [RFC] non-preemptible kernel socket for RAMster References: <4232c4b6-15be-42d8-be42-6e27f9188ce2@default> <6147447c-ecab-43ea-9b4a-1ff64b2089f0@default> <704d094e-7b81-480f-8363-327218d1b0ea@default D3F292ADF945FB49B35E96C94C2061B91257DCA8@nsmail.netscout.com> In-Reply-To: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Sender: owner-linux-mm@kvack.org List-ID: To: "Loke, Chetan" , netdev@vger.kernel.org Cc: Konrad Wilk , linux-mm > From: Loke, Chetan [mailto:Chetan.Loke@netscout.com] > Subject: RE: [RFC] non-preemptible kernel socket for RAMster >=20 > > From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] >=20 > > Actually, RAMster is using a much more flexible type of > > RAM-drive; it is built on top of Transcendent Memory > > and on top of zcache (and thus on top of cleancache and > > frontswap). A RAM-drive is fixed size so is not very suitable > > for the flexibility required for RAMster. For example, > > suppose you have two machines A and B. At one point in > > time A is overcommitted and needs to swap and B is relatively > > idle. Then later, B is overcommitted and needs to swap and > > A is relatively idle. RAMster can handle this entirely > > dynamically, a RAM-drive cannot. >=20 > Again, iff NBD works with a ram-drive then you really wouldn't need to > do anything. How often are you going to re-size your remote-SWAP? Plus, > you can make nbd-server listen on multiple ports - Google(Linux NBD) > returned: http://www.fi.muni.cz/~kripac/orac-nbd/ . Look at the > nbd-server code to see if it launches multiple kernel-threads for > servicing different ports. If not, one can enhance it and scale that way > too. But nbd-server today can service multiple-ports(that is effectively > servicing multiple clients). So why not add NBD-filesystem-filters to > make it point to local/remote swap? Well, we may be talking past each other, but the RAMster answer to: > How often are you going to re-size your remote-SWAP? is "as often as the working set changes on any machine in the cluster", meaning *constantly*, entirely dynamically! How about a more specific example: Suppose you have 2 machines, each with 8GB of memory. 99% of the time each machine is chugging along just fine and doesn't really need more than 4GB, and may even use less than 1GB a large part of the time. But very now and then, one of the machines randomly needs 9GB, 10GB, maybe even 12GB of memory. This would normally result in swapping. (Most system administrators won't even have this much information... they'll just know they are seeing swapping and decide they need to buy more RAM.) With NBD to a ram-drive, each machine would need to pre-allocate 4GB of RAM for the RAM-drive, leaving only 4GB of RAM for the "local" RAM. The result will actually be MORE swapping because a fixed amount of RAM has been pre-reserved for the other machine's swap. With RAMster, everything is done dynamically, so all that matters is the maximum of the sum of the RAM used. You may even be able to *remove* ~2GB of RAM from each of the systems and still never see any swapping to disk. > > Thanks. Could you provide a pointer for this? I found > > the SCST sourceforge page but no obvious references to > > scst-in-ram-mode. (But also, since it appears to be > > SCSI-related, I wonder if it also assumes a fixed size > > target device, RAM or disk or ??) >=20 > Yes, it is SCSI. You should be looking for SCST I/O modes. Read some > docs and then send an email to the scst-mailing-list. If you speak about > block-IO-performance then FC(in its class of price/performance factor) > is more than capable of handling any workload. FC is a protocol designed > for storage. No exotic fabric other than FC is needed. > Folks who start with ethernet for block-IO, always start with bare > minimal code and then for squeezing block-IO performance(aka version 2 > of the product), keep hacking repeatedly or go for a link-speed upgrade. > Start with FC, period. My point was that block I/O devices (AFAIK) always present a fixed "size" to the kernel, and if this is also true of scst-in-ram-mode, the same problem as swap-over-NBD occurs... it's not dynamic. RAMster does not present a block-I/O storage-like interface; it's using the Transcendent Memory interface, which is designed for "slow RAM" of an unknown-and-dynamic size. I'm not a storage expert either, but I do wonder if "no exotic fabric other than FC" isn't an oxymoron ;-) FC is certainly too exotic for me. Dan -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail6.bemta8.messagelabs.com (mail6.bemta8.messagelabs.com [216.82.243.55]) by kanga.kvack.org (Postfix) with ESMTP id AB8C16B004A for ; Wed, 6 Jul 2011 14:12:07 -0400 (EDT) Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Subject: RE: [RFC] non-preemptible kernel socket for RAMster Date: Wed, 6 Jul 2011 14:12:04 -0400 Message-ID: In-Reply-To: References: <4232c4b6-15be-42d8-be42-6e27f9188ce2@default> <6147447c-ecab-43ea-9b4a-1ff64b2089f0@default> <704d094e-7b81-480f-8363-327218d1b0ea@default D3F292ADF945FB49B35E96C94C2061B91257DCA8@nsmail.netscout.com> From: "Loke, Chetan" Sender: owner-linux-mm@kvack.org List-ID: To: Dan Magenheimer , netdev@vger.kernel.org Cc: Konrad Wilk , linux-mm > -----Original Message----- > From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] > Sent: July 05, 2011 9:06 PM > To: Loke, Chetan; netdev@vger.kernel.org > Cc: Konrad Wilk; linux-mm > Subject: RE: [RFC] non-preemptible kernel socket for RAMster >=20 > > From: Loke, Chetan [mailto:Chetan.Loke@netscout.com] > > Subject: RE: [RFC] non-preemptible kernel socket for RAMster > > > > > From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] >=20 > > How often are you going to re-size your remote-SWAP? >=20 > is "as often as the working set changes on any machine in the > cluster", meaning *constantly*, entirely dynamically! How > about a more specific example: Suppose you have 2 machines, > each with 8GB of memory. 99% of the time each machine is > chugging along just fine and doesn't really need more than 4GB, > and may even use less than 1GB a large part of the time. > But very now and then, one of the machines randomly needs > 9GB, 10GB, maybe even 12GB of memory. This would normally > result in swapping. (Most system administrators won't even > have this much information... they'll just know they are > seeing swapping and decide they need to buy more RAM.) >=20 Ok, I understand there is interest in implementing 'remote-volatile-ballooning-variant' but how do you pick a remote candidate(hypervisor)? Let's say, memory could be available on remote system but what if the remote-p{NIC,CPU} is overloaded? Sure, sysadmins won't have this info because this so dynamic(and it's quite possible as you mentioned above). But does the trans-remote-API know about this resource-availability before opening a remote-channel? Stressing the remote-p{NIC/CPU} might trick hypervisor-vmotion-plugin to vmotion VM[s] to another hypervisor. How is trans-remote-API integrating with remote/global vmotion policies to avoid this false vmotion? > Dan Chetan Loke -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail6.bemta12.messagelabs.com (mail6.bemta12.messagelabs.com [216.82.250.247]) by kanga.kvack.org (Postfix) with ESMTP id DAB889000C2 for ; Thu, 7 Jul 2011 11:34:36 -0400 (EDT) MIME-Version: 1.0 Message-ID: <61fd635c-d01c-46c1-ba89-2915a6ddb9f1@default> Date: Thu, 7 Jul 2011 08:34:09 -0700 (PDT) From: Dan Magenheimer Subject: RE: [RFC] non-preemptible kernel socket for RAMster References: <4232c4b6-15be-42d8-be42-6e27f9188ce2@default> <6147447c-ecab-43ea-9b4a-1ff64b2089f0@default> <704d094e-7b81-480f-8363-327218d1b0ea@default> In-Reply-To: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Sender: owner-linux-mm@kvack.org List-ID: To: "Loke, Chetan" , netdev@vger.kernel.org Cc: Konrad Wilk , linux-mm > From: Loke, Chetan [mailto:Chetan.Loke@netscout.com] > Subject: RE: [RFC] non-preemptible kernel socket for RAMster >=20 > > -----Original Message----- > > From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] > > > > > From: Loke, Chetan [mailto:Chetan.Loke@netscout.com] > > > > > > > From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] > > > > > How often are you going to re-size your remote-SWAP? > > > > is "as often as the working set changes on any machine in the > > cluster", meaning *constantly*, entirely dynamically! How > > about a more specific example: Suppose you have 2 machines, > > each with 8GB of memory. 99% of the time each machine is > > chugging along just fine and doesn't really need more than 4GB, > > and may even use less than 1GB a large part of the time. > > But very now and then, one of the machines randomly needs > > 9GB, 10GB, maybe even 12GB of memory. This would normally > > result in swapping. (Most system administrators won't even > > have this much information... they'll just know they are > > seeing swapping and decide they need to buy more RAM.) > > >=20 > Ok, I understand there is interest in implementing > 'remote-volatile-ballooning-variant' but how do you pick a remote > candidate(hypervisor)? Let's say, memory could be available on remote > system but what if the remote-p{NIC,CPU} is overloaded? Sure, sysadmins > won't have this info because this so dynamic(and it's quite possible as > you mentioned above). But does the trans-remote-API know about this > resource-availability before opening a remote-channel? >=20 > Stressing the remote-p{NIC/CPU} might trick hypervisor-vmotion-plugin to > vmotion VM[s] to another hypervisor. How is trans-remote-API integrating > with remote/global vmotion policies to avoid this false vmotion? Hi Chetan -- Thanks for the continued discussion. First, let me clarify that RAMster does not depend on virtualization. At some time in the future, it may be a nice addition for KVM*, but the version I am developing currently only works on a cluster of physical machines. So vmotion/migration is not an issue right now As for choosing the remote machine, another key feature of the Transcendent Memory mechanism is that any and every page can be rejected. If rejected, the page remains local. In essence, on *every* page-to-be-swapped, machine A *asks* machine B, "can you take this page"? If the answer is no, machine A can choose another machine (C), or may choose to swap the page to its own slow swap disk. (Currently, only the latter is implemented, but more complicated policy could certainly be implemented.) Dan * Xen doesn't have drivers so RAMster-over-network is not an option for Xen. A future RAMster-over-exofabric might work with Xen though.) And, by the way, the Transcendent Memory implementation on Xen does handle vmotion/migration so it is a solvable problem. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: email@kvack.org