[Lustre-devel] faking LNET scale

[Lustre-devel] faking LNET scale

[Nicholas Henke]

Greetings -

	I was looking into ways to simulate scale at the LNET level. It would allow us 
to test the LNDs better with less hardware, not to mention things like LNet 
SelfTest and friends.

	With the work in bug 15332 to add multiple nets per NIC, it seemed fairly close 
that we could use that to generate multiple LND connections from a single NIC. 
Ideally we'd have a server or router that would have just one LND instance 
(ptl0) and the client nodes with multiple interfaces (ptl1, ptl2, ...). This 
would increase the load on those server nodes to something interesting.

However, to do this either hacking up lnet_ptlcompat_matchXXX to look at another 
flag besides the_lnet.ln_ptlcompat or some other way of allowing a server with a 
single NET (ptl0) to accept requests from a variety of nets (ptl1, ptl2, etc). 
One cannot use multiple interfaces for the same net type with ln_ptlcompat enabled.

	Is there a better way to do this ? What would be the least abusive of th e rules ?

Cheers,
Nic

[Lustre-devel] faking LNET scale

[Liang Zhen]

Nic,
It's very late night for me now, my head is not clear enough for me to 
make sure whether I'm saying something crazy, :)
LNet always thinks target is remote network(needs router) if it can't 
find a NI with same network ID, for example, if local NI is (ptl0) and 
caller wants to send message to (ptl1), then LNet will:
1. Try to find local  NI for ptl1, and failed then:
2. try to find if ptl1 is a remote network and whether there is router 
for this network (ptl1)

So if you want your server has only one NI instance and can talk with a 
set of different networks, and at the same time, it can talk with other 
remote networks via routers,  I would suggest:
1. create a new command, for example: lctl add_local_net ptl0 ptl[1-N], 
which means LNet should allow NI(ptl0) accessing networks( ptl[1-N] as 
local networks.
2. add a new structure in LNet, i.e:
struct  {
      struct list_head ln_list;
    __u32                ln_net;
     lnet_ni_t          *ln_localni;
     ......
}lnet_localnet_t;
As you see, it's very like current structure lnet_remotenet_t, which is 
pending on lnet_t::ln_remote_nets; we can create a lnet_locallnet_t 
object and add it to global list (i.e: lnet_t::ln_local_nets) by the 
command we mentioned above: lctl add_local_net
3. once upper layer caller sending message, lnet_send() should check 
lnet_t::ln_local_nets firstly (before thinking it's a remote network and 
checking on lnet_t::ln_remote_nets), if it is on 
lnet_t::ln_local_netsthen we can take the local NI. on 
lnet_locanet_t::ln_localni;
4. We need add a new flag for LND, only LND with the flag can support 
command lctl add_local_net.
5. make the LND wouldn't reject messages from different networks.
again, hope I'm answering what you are asking, :)

Regards
Liang

Nicholas Henke wrote:
> Greetings -
>
> 	I was looking into ways to simulate scale at the LNET level. It would allow us 
> to test the LNDs better with less hardware, not to mention things like LNet 
> SelfTest and friends.
>
> 	With the work in bug 15332 to add multiple nets per NIC, it seemed fairly close 
> that we could use that to generate multiple LND connections from a single NIC. 
> Ideally we'd have a server or router that would have just one LND instance 
> (ptl0) and the client nodes with multiple interfaces (ptl1, ptl2, ...). This 
> would increase the load on those server nodes to something interesting.
>
> However, to do this either hacking up lnet_ptlcompat_matchXXX to look at another 
> flag besides the_lnet.ln_ptlcompat or some other way of allowing a server with a 
> single NET (ptl0) to accept requests from a variety of nets (ptl1, ptl2, etc). 
> One cannot use multiple interfaces for the same net type with ln_ptlcompat enabled.
>
> 	Is there a better way to do this ? What would be the least abusive of th e rules ?
>
> Cheers,
> Nic
> _______________________________________________
> Lustre-devel mailing list
> Lustre-devel at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-devel
>   

[Lustre-devel] faking LNET scale

[Nic Henke]

Liang Zhen wrote:
> Nic,
> It's very late night for me now, my head is not clear enough for me to
> make sure whether I'm saying something crazy, :)
Liang,
    Thanks for the notes - at worst this is crazy interesting :-) This
looks very doable - I've not dug into the code to see if there are any
implementation gotchas - but it looks like it should work. I'll let you
know what I come up with.

Cheers,
Nic

> LNet always thinks target is remote network(needs router) if it can't
> find a NI with same network ID, for example, if local NI is (ptl0) and
> caller wants to send message to (ptl1), then LNet will:
> 1. Try to find local  NI for ptl1, and failed then:
> 2. try to find if ptl1 is a remote network and whether there is router
> for this network (ptl1)
>
> So if you want your server has only one NI instance and can talk with
> a set of different networks, and at the same time, it can talk with
> other remote networks via routers,  I would suggest:
> 1. create a new command, for example: lctl add_local_net ptl0
> ptl[1-N], which means LNet should allow NI(ptl0) accessing networks(
> ptl[1-N] as local networks.
> 2. add a new structure in LNet, i.e:
> struct  {
>      struct list_head ln_list;
>    __u32                ln_net;
>     lnet_ni_t          *ln_localni;
>     ......
> }lnet_localnet_t;
> As you see, it's very like current structure lnet_remotenet_t, which
> is pending on lnet_t::ln_remote_nets; we can create a lnet_locallnet_t
> object and add it to global list (i.e: lnet_t::ln_local_nets) by the
> command we mentioned above: lctl add_local_net
> 3. once upper layer caller sending message, lnet_send() should check
> lnet_t::ln_local_nets firstly (before thinking it's a remote network
> and checking on lnet_t::ln_remote_nets), if it is on
> lnet_t::ln_local_netsthen we can take the local NI. on
> lnet_locanet_t::ln_localni;
> 4. We need add a new flag for LND, only LND with the flag can support
> command lctl add_local_net.
> 5. make the LND wouldn't reject messages from different networks.
> again, hope I'm answering what you are asking, :)
>
> Regards
> Liang

[Lustre-devel] faking LNET scale

[Eric Barton]

Why not just instantiate all the NIs on the server?  LNDs that
support multiple NIs typically have a single set of global tables,
so it should still stress the LND just fine.  Also having n different
targets (one for each LNET) on the server actually simplifies
client configuration too - if you only have a single target, lustre
would, by default, only use one client NID to get to it.


    Cheers,
              Eric



> -----Original Message-----
> From: lustre-devel-bounces at lists.lustre.org [mailto:lustre-devel-bounces at lists.lustre.org] On Behalf Of Liang Zhen
> Sent: 17 April 2009 7:34 PM
> To: Nicholas Henke
> Cc: lustre-devel at lists.lustre.org
> Subject: Re: [Lustre-devel] faking LNET scale
> 
> Nic,
> It's very late night for me now, my head is not clear enough for me to
> make sure whether I'm saying something crazy, :)
> LNet always thinks target is remote network(needs router) if it can't
> find a NI with same network ID, for example, if local NI is (ptl0) and
> caller wants to send message to (ptl1), then LNet will:
> 1. Try to find local  NI for ptl1, and failed then:
> 2. try to find if ptl1 is a remote network and whether there is router
> for this network (ptl1)
> 
> So if you want your server has only one NI instance and can talk with a
> set of different networks, and at the same time, it can talk with other
> remote networks via routers,  I would suggest:
> 1. create a new command, for example: lctl add_local_net ptl0 ptl[1-N],
> which means LNet should allow NI(ptl0) accessing networks( ptl[1-N] as
> local networks.
> 2. add a new structure in LNet, i.e:
> struct  {
>       struct list_head ln_list;
>     __u32                ln_net;
>      lnet_ni_t          *ln_localni;
>      ......
> }lnet_localnet_t;
> As you see, it's very like current structure lnet_remotenet_t, which is
> pending on lnet_t::ln_remote_nets; we can create a lnet_locallnet_t
> object and add it to global list (i.e: lnet_t::ln_local_nets) by the
> command we mentioned above: lctl add_local_net
> 3. once upper layer caller sending message, lnet_send() should check
> lnet_t::ln_local_nets firstly (before thinking it's a remote network and
> checking on lnet_t::ln_remote_nets), if it is on
> lnet_t::ln_local_netsthen we can take the local NI. on
> lnet_locanet_t::ln_localni;
> 4. We need add a new flag for LND, only LND with the flag can support
> command lctl add_local_net.
> 5. make the LND wouldn't reject messages from different networks.
> again, hope I'm answering what you are asking, :)
> 
> Regards
> Liang
> 
> Nicholas Henke wrote:
> > Greetings -
> >
> > 	I was looking into ways to simulate scale at the LNET level. It would allow us
> > to test the LNDs better with less hardware, not to mention things like LNet
> > SelfTest and friends.
> >
> > 	With the work in bug 15332 to add multiple nets per NIC, it seemed fairly close
> > that we could use that to generate multiple LND connections from a single NIC.
> > Ideally we'd have a server or router that would have just one LND instance
> > (ptl0) and the client nodes with multiple interfaces (ptl1, ptl2, ...). This
> > would increase the load on those server nodes to something interesting.
> >
> > However, to do this either hacking up lnet_ptlcompat_matchXXX to look at another
> > flag besides the_lnet.ln_ptlcompat or some other way of allowing a server with a
> > single NET (ptl0) to accept requests from a variety of nets (ptl1, ptl2, etc).
> > One cannot use multiple interfaces for the same net type with ln_ptlcompat enabled.
> >
> > 	Is there a better way to do this ? What would be the least abusive of th e rules ?
> >
> > Cheers,
> > Nic
> > _______________________________________________
> > Lustre-devel mailing list
> > Lustre-devel at lists.lustre.org
> > http://lists.lustre.org/mailman/listinfo/lustre-devel
> >
> 
> _______________________________________________
> Lustre-devel mailing list
> Lustre-devel at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-devel

[Lustre-devel] faking LNET scale

[Nicholas Henke]

Liang Zhen wrote:
> Nic,
> It's very late night for me now, my head is not clear enough for me to 
> make sure whether I'm saying something crazy, :)
> LNet always thinks target is remote network(needs router) if it can't 
> find a NI with same network ID, for example, if local NI is (ptl0) and 
> caller wants to send message to (ptl1), then LNet will:
> 1. Try to find local  NI for ptl1, and failed then:
> 2. try to find if ptl1 is a remote network and whether there is router 
> for this network (ptl1)
> 
> So if you want your server has only one NI instance and can talk with a 
> set of different networks, and at the same time, it can talk with other 
> remote networks via routers,  I would suggest:
> 1. create a new command, for example: lctl add_local_net ptl0 ptl[1-N], 
> which means LNet should allow NI(ptl0) accessing networks( ptl[1-N] as 
> local networks.
> 2. add a new structure in LNet, i.e:
> struct  {
>      struct list_head ln_list;
>    __u32                ln_net;
>     lnet_ni_t          *ln_localni;
>     ......
> }lnet_localnet_t;
> As you see, it's very like current structure lnet_remotenet_t, which is 
> pending on lnet_t::ln_remote_nets; we can create a lnet_locallnet_t 
> object and add it to global list (i.e: lnet_t::ln_local_nets) by the 
> command we mentioned above: lctl add_local_net
> 3. once upper layer caller sending message, lnet_send() should check 
> lnet_t::ln_local_nets firstly (before thinking it's a remote network and 
> checking on lnet_t::ln_remote_nets), if it is on 
> lnet_t::ln_local_netsthen we can take the local NI. on 
> lnet_locanet_t::ln_localni;
> 4. We need add a new flag for LND, only LND with the flag can support 
> command lctl add_local_net.
> 5. make the LND wouldn't reject messages from different networks.
> again, hope I'm answering what you are asking, :)

This is almost working - I'm running into one problem: lnet_accept wants to 
match the ni->ni_nid against the requested NID. It is failing as the nets don't 
match (ptl1 vs ptl0).

It looks like there are a fair number of places like this, most using 
lnet_ptlcompat_match{net,nid}.

How should I handle those? Add another clause like ptlcompat (like ln_aliases) 
and if that is set (we have aliases set), do a search to find the alias and see 
if there is an alias that would allow NIDNET(lnet_net) == NIDNET(ptl_net)?

Is there a cleaner way?

Nic

[Lustre-devel] faking LNET scale

[Liang Zhen]

Hi Nic,
For incoming requests, I think we can share the same network aliases 
with outgoing messsages (i.e: lnet_t::ln_local_nets in my previous 
mail), matching on the aliases list could be embedded in 
lnet_ptlcompat_match{net,nid} and lnet_net2ni_locked so we don't need 
worry about changing code everywhere.

Regards
Liang

Nicholas Henke wrote:
> Liang Zhen wrote:
>> Nic,
>> It's very late night for me now, my head is not clear enough for me 
>> to make sure whether I'm saying something crazy, :)
>> LNet always thinks target is remote network(needs router) if it can't 
>> find a NI with same network ID, for example, if local NI is (ptl0) 
>> and caller wants to send message to (ptl1), then LNet will:
>> 1. Try to find local  NI for ptl1, and failed then:
>> 2. try to find if ptl1 is a remote network and whether there is 
>> router for this network (ptl1)
>>
>> So if you want your server has only one NI instance and can talk with 
>> a set of different networks, and at the same time, it can talk with 
>> other remote networks via routers,  I would suggest:
>> 1. create a new command, for example: lctl add_local_net ptl0 
>> ptl[1-N], which means LNet should allow NI(ptl0) accessing networks( 
>> ptl[1-N] as local networks.
>> 2. add a new structure in LNet, i.e:
>> struct  {
>>      struct list_head ln_list;
>>    __u32                ln_net;
>>     lnet_ni_t          *ln_localni;
>>     ......
>> }lnet_localnet_t;
>> As you see, it's very like current structure lnet_remotenet_t, which 
>> is pending on lnet_t::ln_remote_nets; we can create a 
>> lnet_locallnet_t object and add it to global list (i.e: 
>> lnet_t::ln_local_nets) by the command we mentioned above: lctl 
>> add_local_net
>> 3. once upper layer caller sending message, lnet_send() should check 
>> lnet_t::ln_local_nets firstly (before thinking it's a remote network 
>> and checking on lnet_t::ln_remote_nets), if it is on 
>> lnet_t::ln_local_netsthen we can take the local NI. on 
>> lnet_locanet_t::ln_localni;
>> 4. We need add a new flag for LND, only LND with the flag can support 
>> command lctl add_local_net.
>> 5. make the LND wouldn't reject messages from different networks.
>> again, hope I'm answering what you are asking, :)
>
> This is almost working - I'm running into one problem: lnet_accept 
> wants to match the ni->ni_nid against the requested NID. It is failing 
> as the nets don't match (ptl1 vs ptl0).
>
> It looks like there are a fair number of places like this, most using 
> lnet_ptlcompat_match{net,nid}.
>
> How should I handle those? Add another clause like ptlcompat (like 
> ln_aliases) and if that is set (we have aliases set), do a search to 
> find the alias and see if there is an alias that would allow 
> NIDNET(lnet_net) == NIDNET(ptl_net)?
>
> Is there a cleaner way?
>
> Nic

[Lustre-devel] faking LNET scale

[Isaac Huang]

On Fri, Apr 17, 2009 at 12:10:01PM -0500, Nicholas Henke wrote:
> ......
> However, to do this either hacking up lnet_ptlcompat_matchXXX to look at another 
> flag besides the_lnet.ln_ptlcompat or some other way of allowing a server with a 
> single NET (ptl0) to accept requests from a variety of nets (ptl1, ptl2, etc). 
> One cannot use multiple interfaces for the same net type with ln_ptlcompat enabled.

Note that Portals compatibility (lnet_ptlcompat_, the_lnet.ln_ptlcompat, and friends)
have already been removed from lnet HEAD, on which all 2.x and future releases will 
be based.

> 	Is there a better way to do this ? What would be the least abusive of th e rules ?

If you only have limited number of test nodes, one way to drive the
network as hard as possible is to have all nodes use a very high
ptllnd peercredits option and run LST test with a high concurrency
(with the latest LST patch from 15332).

Thanks,
Isaac

[Lustre-devel] faking LNET scale

[Nic Henke]

Isaac Huang wrote:
> On Fri, Apr 17, 2009 at 12:10:01PM -0500, Nicholas Henke wrote:
>   
>> ......
>> However, to do this either hacking up lnet_ptlcompat_matchXXX to look at another 
>> flag besides the_lnet.ln_ptlcompat or some other way of allowing a server with a 
>> single NET (ptl0) to accept requests from a variety of nets (ptl1, ptl2, etc). 
>> One cannot use multiple interfaces for the same net type with ln_ptlcompat enabled.
>>     
>
> Note that Portals compatibility (lnet_ptlcompat_, the_lnet.ln_ptlcompat, and friends)
> have already been removed from lnet HEAD, on which all 2.x and future releases will 
> be based.
>
>   
>> 	Is there a better way to do this ? What would be the least abusive of th e rules ?
>>     
>
> If you only have limited number of test nodes, one way to drive the
> network as hard as possible is to have all nodes use a very high
> ptllnd peercredits option and run LST test with a high concurrency
> (with the latest LST patch from 15332).
>
> Thanks,
> Isaac
>   
I was more interested in scaling the number of peers/connections. The
previous suggestion about doing a localnet check would help do that. 
From past experience, we don't often find too many issues just getting
the data moving when changing to higher scale - it is all the mgmt of
peers/connections that end up getting 'fun'.

As you say - just using higher credits is usually sufficient to max out
the network throughput for a given set of nodes.


Nic