Re: Scheduled Transfer Protocol on Linux

Re: Scheduled Transfer Protocol on Linux

[Larry McVoy]

: > OK, so tell me: how many locks does it take to scale up the following to
: > 16 CPUs:
: > 	local disks
: > 	local file system
: > 	remote file system
: > 	processes
: > 	networking interfaces and stack
: >
: > What do the locks cover?  At 16 CPUs, can you keep all the locks straight
: > in your head?  Nope.  So what happens when you go into the kernel and add
: > a feature?  You add a lock.  What does that do?  Increases the number of
: > locks.  What effect does that have?  Makes it more likely that you'll add
: > more locks, because now it is even less obvious what the lock protects.
: 
: Good design can avoid these problems.  If it isn't obvious what a lock 
: protects, you should rethink your locking structure.

I notice that you didn't actually answer any of the questions above, just
a nice hand wave that says "it need not be so".  Well, just to indulge
me, could you please either:

    a) answer the questions above, or
    b) show a shipping, production system which demonstrates your claims, or
    c) admit that you don't know the answer.

: I wasn't arguing that 16 way SMP is OK.  Everyone knows it isn't.

Geez, and this from the guy who said Linux needs to support "16-64 way SMP".

: Are you saying that clusters of small SMP machines are better?  

read these:
	http://www.bitmover.com/llnl/smp.pdf
	http://www.bitmover.com/llnl/labs.pdf

: So the locking 
: moves from the kernels to the application layer.  You still have the same 
: synchronization concerns, it's just a matter of what layer they are 
: implemented at.

Err, if you had actually done this, you'd find that your statements
are unsupportable in practice.  Please show me an application that has
anything, even with an order of magnitude, like the number of locks
taken/released per second in IRIX or Solaris.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/

Re: Scheduled Transfer Protocol on Linux

[Aman Singla]

I think Larry's description (below) about how STP works is pretty
much accurate modulo 'various levels of details / mechanisms' but
hey, thats what protocol spec's are for :-)
http://www.hippi.org/cST.html

The STP Long Message Transfer stuff is realized to be useful for
network enabled storage - may it be through higher performance
solutions in the upper layer - NFS/BDS etc., or via the lower layers
like SCSI over STP.

What I'd like to add is, that STP also specifies persistent (longer
duration - spanning multiple messages) mappings and pinning down of
memory which is useful for low latency communications a'la clustering
applications. Again, what STP offers here is an open, media-independent
and scalable protocol.

Are there any other applications that people can forsee?

thanks,

:a

> What it does do is give you a high bandwidth, low CPU cycle way of moving
> the data.
> 
> The simplified view of STP is really just remote DMA.  How it works, again
> greatly simplified, is sort of like this:
> 
>         client                                  server
>         request to send 1GB ->
> 
>                         <- please wait while I set up some pages (optional)
> 
>                         <- clear to send first 100MB of data, here's a cookie
> 
>         data on wire, prefixed with incrementing cookie ->
>         data on wire, prefixed with incrementing cookie ->
>         data on wire, prefixed with incrementing cookie ->
>         data on wire, prefixed with incrementing cookie ->
>         data on wire, prefixed with incrementing cookie ->
>         data on wire, prefixed with incrementing cookie ->
>         data on wire, prefixed with incrementing cookie ->
>         data on wire, prefixed with incrementing cookie ->
>         data on wire, prefixed with incrementing cookie ->
> 
>                         -> data hits NIC, cockie is index into CAM, exactly
>                            like an ethernet switch indexes MAC addresses,
>                            except that the value here is a physical page
>                            address instead of an outgoing port
> 
>                         -> data hits NIC ....
>                         -> data hits NIC ....
>                         -> data hits NIC ....
>                         -> data hits NIC ....
>                         -> data hits NIC ....
>                         -> data hits NIC .... we're at 75MB or so
> 
>                            server goes off and locks down another 100MB
> 
>                         <- CTS another 100MB (this happens well before the
>                            first 100MB is drained, so there are no bubbles,
>                            the pipe is 100% full at all times)
> 
> I trust that the SGI guys, who know this stuff better than I, will correct
> and problems with this description.  I'm sure there are some, but I'm also
> sure this is pretty close to how it works modulo flow control, security,
> and other important bits.
> 
> What's important is that you notice two things:
> 
>     (a) this is what I call "polite networking".  Normal networking is
>         impolite, the packets show up on your doorstep uninvited.  Here,
>         the receiver is nicely asked if it is OK, and gets to set things
>         up first.  Just like in DMA.
> 
>     (b) There is some hardware on the NIC that translates from packets to
>         physical page addresses, this happens inline, so the packet gets
>         DMAed into memory as it is coming off the wire.  Very low buffering
>         requirements.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/