SATA "target mode" (or "Channel-to-Channel" comm mode)

SATA "target mode" (or "Channel-to-Channel" comm mode)

[Mark Lord]

Guys,

Within the next couple of weeks, I would like to submit patches
for 2.6.30 for a simple form of what Marvell likes to call "target mode",
or C2C (channel-to-channel communications).

This is for sata_mv.

The question is, how to expose an interface to actually access it?

Quick background on Marvell C2C:

1. C2C is only for Gen2 and Gen2e chipsets.

2. Requires a special SATA cross-over (simple twist) cable
    between two SATA ports.  Ports can be on the same host
    adaptor or on separate adaptors and/or machines.

3. Each sata_mv port can be either a (0) normal SATA host,
    or (1) special SATA C2C initiator, or (2) a SATA target device.

4. A Gen2e mode (2) target can connect/communicate with either
    a mode (0) host or a mode (1) initiator.  I'm not yet sure
    whether an older Gen2 target can connect with a mode (0) host.

5. Mode (1) initiator appears to relax requirements such as waiting
    for a device BUSY bit to clear etc., and is intended for simple
    channel-to-channel communications.

6. A boot/module parameter seems to be the best way to enable
    this feature, as otherwise libata wastes a lot of time and
    effort probing for non-existant drives and slowing down
    the boot process.

7. Initially, all that we want is a way to use two SATA ports
    (on the same or different machines) as a simple byte-stream
    communications channel, between a mode (1) inititiator
    and a mode (2) target.  This is used in real-life as a high-speed
    local comm channel between halves of split server machines.

8. Transparently emulating a SATA drive is possible on Gen2e chips
    at least, and perhaps also on Gen2 chips.  This is not being
    worked on at this time.

9. Using two ports in tandem, one mode (0) host and one mode (2) target,
    one can construct a quite capable SATA capture/analyzer device
    which could be inserted in between any other SATA host and device.
    Quite useful, and something I intend to work on later this year.

So, starting with simple stuff, I want to expose an interface for
point 7 above.  The thought is to use netlink for this, on both ends.

An alternative might be to tie it into the SCSI Target Framework (tgt).
But that is more for full target device emulation than for simple comms.
And SATA is not SCSI, so it could really restrict/prevent us from doing
a full SATA emulation (eg. point 9) in the end.

Time is short, so I'd like to spend it on something that Jeff would
actually accept.  Thus this email.

Jeff?

Re: SATA "target mode" (or "Channel-to-Channel" comm mode)

[Jeff Garzik]

Mark Lord wrote:
> Guys,
> 
> Within the next couple of weeks, I would like to submit patches
> for 2.6.30 for a simple form of what Marvell likes to call "target mode",
> or C2C (channel-to-channel communications).
> 
> This is for sata_mv.
> 
> The question is, how to expose an interface to actually access it?
> 
> Quick background on Marvell C2C:
> 
> 1. C2C is only for Gen2 and Gen2e chipsets.
> 
> 2. Requires a special SATA cross-over (simple twist) cable
>    between two SATA ports.  Ports can be on the same host
>    adaptor or on separate adaptors and/or machines.
> 
> 3. Each sata_mv port can be either a (0) normal SATA host,
>    or (1) special SATA C2C initiator, or (2) a SATA target device.
> 
> 4. A Gen2e mode (2) target can connect/communicate with either
>    a mode (0) host or a mode (1) initiator.  I'm not yet sure
>    whether an older Gen2 target can connect with a mode (0) host.
> 
> 5. Mode (1) initiator appears to relax requirements such as waiting
>    for a device BUSY bit to clear etc., and is intended for simple
>    channel-to-channel communications.
> 
> 6. A boot/module parameter seems to be the best way to enable
>    this feature, as otherwise libata wastes a lot of time and
>    effort probing for non-existant drives and slowing down
>    the boot process.
> 
> 7. Initially, all that we want is a way to use two SATA ports
>    (on the same or different machines) as a simple byte-stream
>    communications channel, between a mode (1) inititiator
>    and a mode (2) target.  This is used in real-life as a high-speed
>    local comm channel between halves of split server machines.
> 
> 8. Transparently emulating a SATA drive is possible on Gen2e chips
>    at least, and perhaps also on Gen2 chips.  This is not being
>    worked on at this time.
> 
> 9. Using two ports in tandem, one mode (0) host and one mode (2) target,
>    one can construct a quite capable SATA capture/analyzer device
>    which could be inserted in between any other SATA host and device.
>    Quite useful, and something I intend to work on later this year.
> 
> So, starting with simple stuff, I want to expose an interface for
> point 7 above.  The thought is to use netlink for this, on both ends.
> 
> An alternative might be to tie it into the SCSI Target Framework (tgt).
> But that is more for full target device emulation than for simple comms.
> And SATA is not SCSI, so it could really restrict/prevent us from doing
> a full SATA emulation (eg. point 9) in the end.
> 
> Time is short, so I'd like to spend it on something that Jeff would
> actually accept.  Thus this email.

It depends on the task.

The miscdev (i.e. chrdev) interface found in drivers/scsi/scsi_tgt_if.c 
of repo [1] seems pretty generic, simple, small and applicable to 
portions of the problem presented here...  The basic task in 
scsi_tgt_if's case is just shoveling packets to/from userspace.

The interface is going to depend on the implementations that wind up in 
the field, for each task.

SATA packet capture:  highly useful, but implies _copying_ the packets 
before passing them on to regular channels.  So, does this imply packets 
will be copied kernel->userspace->kernel ?  kernel->kernel?  The 
interface will be vastly different in each case.

SATA target:  again, userspace or kernel?  Presumably scsi_tgt_if.c is 
applicable to this task.

Also, in general, it sounds like we need a general way to put a port 
into a specific mode (initiator, target, vendor special) during runtime. 
  Then create a module parameter that allows boot-time initialization of 
this port mode selector.

	Jeff



[1] 
http://www.kernel.org/git/?p=linux/kernel/git/tomo/linux-2.6-target.git;a=summary

Re: SATA "target mode" (or "Channel-to-Channel" comm mode)

[Mark Lord]

Jeff Garzik wrote:
> Mark Lord wrote:
>> Guys,
>>
>> Within the next couple of weeks, I would like to submit patches
>> for 2.6.30 for a simple form of what Marvell likes to call "target mode",
>> or C2C (channel-to-channel communications).
>>
>> This is for sata_mv.
>>
>> The question is, how to expose an interface to actually access it?
>>
>> Quick background on Marvell C2C:
>>
>> 1. C2C is only for Gen2 and Gen2e chipsets.
>>
>> 2. Requires a special SATA cross-over (simple twist) cable
>>    between two SATA ports.  Ports can be on the same host
>>    adaptor or on separate adaptors and/or machines.
>>
>> 3. Each sata_mv port can be either a (0) normal SATA host,
>>    or (1) special SATA C2C initiator, or (2) a SATA target device.
>>
>> 4. A Gen2e mode (2) target can connect/communicate with either
>>    a mode (0) host or a mode (1) initiator.  I'm not yet sure
>>    whether an older Gen2 target can connect with a mode (0) host.
>>
>> 5. Mode (1) initiator appears to relax requirements such as waiting
>>    for a device BUSY bit to clear etc., and is intended for simple
>>    channel-to-channel communications.
>>
>> 6. A boot/module parameter seems to be the best way to enable
>>    this feature, as otherwise libata wastes a lot of time and
>>    effort probing for non-existant drives and slowing down
>>    the boot process.
>>
>> 7. Initially, all that we want is a way to use two SATA ports
>>    (on the same or different machines) as a simple byte-stream
>>    communications channel, between a mode (1) inititiator
>>    and a mode (2) target.  This is used in real-life as a high-speed
>>    local comm channel between halves of split server machines.
>>
>> 8. Transparently emulating a SATA drive is possible on Gen2e chips
>>    at least, and perhaps also on Gen2 chips.  This is not being
>>    worked on at this time.
>>
>> 9. Using two ports in tandem, one mode (0) host and one mode (2) target,
>>    one can construct a quite capable SATA capture/analyzer device
>>    which could be inserted in between any other SATA host and device.
>>    Quite useful, and something I intend to work on later this year.
>>
>> So, starting with simple stuff, I want to expose an interface for
>> point 7 above.  The thought is to use netlink for this, on both ends.
>>
>> An alternative might be to tie it into the SCSI Target Framework (tgt).
>> But that is more for full target device emulation than for simple comms.
>> And SATA is not SCSI, so it could really restrict/prevent us from doing
>> a full SATA emulation (eg. point 9) in the end.
>>
>> Time is short, so I'd like to spend it on something that Jeff would
>> actually accept.  Thus this email.
> 
> It depends on the task.
> 
> The miscdev (i.e. chrdev) interface found in drivers/scsi/scsi_tgt_if.c 
> of repo [1] seems pretty generic, simple, small and applicable to 
> portions of the problem presented here...  The basic task in 
> scsi_tgt_if's case is just shoveling packets to/from userspace.
..

Except it's rather SCSI specific, and the userspace frontend even more so.
The code expects a SCSI command block, LUN, TAG, and other fields that
a SATA FIS won't have.  Seems clumsy, particularly when we (in theory)
are trying to decouple libata from SCSI.   But if that's the way,
then I can clumsily wrap each FIS in a fake ATA_16 header or something.

> SATA packet capture:  highly useful, but implies _copying_ the packets 
> before passing them on to regular channels.  So, does this imply packets 
> will be copied kernel->userspace->kernel ?  kernel->kernel?  The 
> interface will be vastly different in each case.
..

Absolutely, which is why I'm leaving that for (much) later,
and looking for a simple (userspace) comms method to begin with,
using the special (non-SATA compliant) "initiator" on one end,
and a "target" on the other end.  This is different from the 
SATA interceptor/emulator configurations.

..
> Also, in general, it sounds like we need a general way to put a port 
> into a specific mode (initiator, target, vendor special) during runtime. 
>  Then create a module parameter that allows boot-time initialization of 
> this port mode selector.
..

That part's easy, and already implemented:  module parameter + sysfs attrs.

Cheers

Re: SATA "target mode" (or "Channel-to-Channel" comm mode)

[Jeff Garzik]

Mark Lord wrote:
> Jeff Garzik wrote:
>> Mark Lord wrote:
>>> Guys,
>>>
>>> Within the next couple of weeks, I would like to submit patches
>>> for 2.6.30 for a simple form of what Marvell likes to call "target 
>>> mode",
>>> or C2C (channel-to-channel communications).
>>>
>>> This is for sata_mv.
>>>
>>> The question is, how to expose an interface to actually access it?
>>>
>>> Quick background on Marvell C2C:
>>>
>>> 1. C2C is only for Gen2 and Gen2e chipsets.
>>>
>>> 2. Requires a special SATA cross-over (simple twist) cable
>>>    between two SATA ports.  Ports can be on the same host
>>>    adaptor or on separate adaptors and/or machines.
>>>
>>> 3. Each sata_mv port can be either a (0) normal SATA host,
>>>    or (1) special SATA C2C initiator, or (2) a SATA target device.
>>>
>>> 4. A Gen2e mode (2) target can connect/communicate with either
>>>    a mode (0) host or a mode (1) initiator.  I'm not yet sure
>>>    whether an older Gen2 target can connect with a mode (0) host.
>>>
>>> 5. Mode (1) initiator appears to relax requirements such as waiting
>>>    for a device BUSY bit to clear etc., and is intended for simple
>>>    channel-to-channel communications.
>>>
>>> 6. A boot/module parameter seems to be the best way to enable
>>>    this feature, as otherwise libata wastes a lot of time and
>>>    effort probing for non-existant drives and slowing down
>>>    the boot process.
>>>
>>> 7. Initially, all that we want is a way to use two SATA ports
>>>    (on the same or different machines) as a simple byte-stream
>>>    communications channel, between a mode (1) inititiator
>>>    and a mode (2) target.  This is used in real-life as a high-speed
>>>    local comm channel between halves of split server machines.
>>>
>>> 8. Transparently emulating a SATA drive is possible on Gen2e chips
>>>    at least, and perhaps also on Gen2 chips.  This is not being
>>>    worked on at this time.
>>>
>>> 9. Using two ports in tandem, one mode (0) host and one mode (2) target,
>>>    one can construct a quite capable SATA capture/analyzer device
>>>    which could be inserted in between any other SATA host and device.
>>>    Quite useful, and something I intend to work on later this year.
>>>
>>> So, starting with simple stuff, I want to expose an interface for
>>> point 7 above.  The thought is to use netlink for this, on both ends.
>>>
>>> An alternative might be to tie it into the SCSI Target Framework (tgt).
>>> But that is more for full target device emulation than for simple comms.
>>> And SATA is not SCSI, so it could really restrict/prevent us from doing
>>> a full SATA emulation (eg. point 9) in the end.
>>>
>>> Time is short, so I'd like to spend it on something that Jeff would
>>> actually accept.  Thus this email.
>>
>> It depends on the task.
>>
>> The miscdev (i.e. chrdev) interface found in 
>> drivers/scsi/scsi_tgt_if.c of repo [1] seems pretty generic, simple, 
>> small and applicable to portions of the problem presented here...  The 
>> basic task in scsi_tgt_if's case is just shoveling packets to/from 
>> userspace.
> ..
> 
> Except it's rather SCSI specific, and the userspace frontend even more so.
> The code expects a SCSI command block, LUN, TAG, and other fields that
> a SATA FIS won't have.  Seems clumsy, particularly when we (in theory)
> are trying to decouple libata from SCSI.   But if that's the way,
> then I can clumsily wrap each FIS in a fake ATA_16 header or something.

The kernel bits are not terribly SCSI-specific.  I'd ignore the userland 
bits.

We just have to get the interface right.  Even if we "cp scsi_tgt_if.c 
ata_tgt_if.c" and then go from there, we should make an effort to create 
similar packet push/pull interfaces.

Creating something wildly different would be counterproductive.


>> SATA packet capture:  highly useful, but implies _copying_ the packets 
>> before passing them on to regular channels.  So, does this imply 
>> packets will be copied kernel->userspace->kernel ?  kernel->kernel?  
>> The interface will be vastly different in each case.
> ..
> 
> Absolutely, which is why I'm leaving that for (much) later,
> and looking for a simple (userspace) comms method to begin with,
> using the special (non-SATA compliant) "initiator" on one end,
> and a "target" on the other end.  This is different from the SATA 
> interceptor/emulator configurations.

I would just bang out an interface.  Userspace interfaces always wind up 
needing some tweaking.

	Jeff

Re: SATA "target mode" (or "Channel-to-Channel" comm mode)

[Mark Lord]

Jeff Garzik wrote:
> Mark Lord wrote:
>> Jeff Garzik wrote:
..
>>> The miscdev (i.e. chrdev) interface found in 
>>> drivers/scsi/scsi_tgt_if.c of repo [1] seems pretty generic, simple, 
>>> small and applicable to portions of the problem presented here...  
>>> The basic task in scsi_tgt_if's case is just shoveling packets 
>>> to/from userspace.
>> ..
>>
>> Except it's rather SCSI specific, and the userspace frontend even more so.
>> The code expects a SCSI command block, LUN, TAG, and other fields that
>> a SATA FIS won't have.  Seems clumsy, particularly when we (in theory)
>> are trying to decouple libata from SCSI.   But if that's the way,
>> then I can clumsily wrap each FIS in a fake ATA_16 header or something.
> 
> The kernel bits are not terribly SCSI-specific.  I'd ignore the userland 
> bits.
> 
> We just have to get the interface right.  Even if we "cp scsi_tgt_if.c 
> ata_tgt_if.c" and then go from there, we should make an effort to create 
> similar packet push/pull interfaces.
..

Yeah, that does look like a good starting point.
To begin with, I'll see if I can just wrap the SATA FIS+data
into something that scsi_tgt_if.c will pass through unmolested.

This will give us a way to play with it in the 2.6.30 time frame,
and we can fork it or tweak it as needed for later releases then.

> Creating something wildly different would be counterproductive.
..

Yes we definitely should re-use that stuff if possible.
I think it all feeds to/from netlink, too, though it's been
a month since I last tried to find my way through it all.

Patches in a week or so.

Cheers

Re: SATA "target mode" (or "Channel-to-Channel" comm mode)

[James Bottomley]

On Tue, 2009-03-03 at 09:37 -0500, Mark Lord wrote:
> Jeff Garzik wrote:
> > Mark Lord wrote:
> >> Time is short, so I'd like to spend it on something that Jeff would
> >> actually accept.  Thus this email.
> > 
> > It depends on the task.
> > 
> > The miscdev (i.e. chrdev) interface found in drivers/scsi/scsi_tgt_if.c 
> > of repo [1] seems pretty generic, simple, small and applicable to 
> > portions of the problem presented here...  The basic task in 
> > scsi_tgt_if's case is just shoveling packets to/from userspace.
> ..
> 
> Except it's rather SCSI specific, and the userspace frontend even more so.
> The code expects a SCSI command block, LUN, TAG, and other fields that
> a SATA FIS won't have.  Seems clumsy, particularly when we (in theory)
> are trying to decouple libata from SCSI.   But if that's the way,
> then I can clumsily wrap each FIS in a fake ATA_16 header or something.

Yes, but SG_IO began life as being entirely SCSI specific, but it's now
the block packet infrastructure.  I don't think anyone has an objection
to the SCSI target infrastructure becoming block target
infrastructure ... the fields can be made optional or more generic to
further this.  Probably it should speak a generic packet with a protocol
identifier and let the user space sort out the details.

James

Re: SATA "target mode" (or "Channel-to-Channel" comm mode)

[Jeff Garzik]

James Bottomley wrote:
> Yes, but SG_IO began life as being entirely SCSI specific, but it's now
> the block packet infrastructure.  I don't think anyone has an objection
> to the SCSI target infrastructure becoming block target
> infrastructure ... the fields can be made optional or more generic to
> further this.  Probably it should speak a generic packet with a protocol
> identifier and let the user space sort out the details.

Agreed...

	Jeff

Re: SATA "target mode" (or "Channel-to-Channel" comm mode)

[FUJITA Tomonori]

On Tue, 03 Mar 2009 16:16:12 +0000
James Bottomley <James.Bottomley@HansenPartnership.com> wrote:

> On Tue, 2009-03-03 at 09:37 -0500, Mark Lord wrote:
> > Jeff Garzik wrote:
> > > Mark Lord wrote:
> > >> Time is short, so I'd like to spend it on something that Jeff would
> > >> actually accept.  Thus this email.
> > > 
> > > It depends on the task.
> > > 
> > > The miscdev (i.e. chrdev) interface found in drivers/scsi/scsi_tgt_if.c 
> > > of repo [1] seems pretty generic, simple, small and applicable to 
> > > portions of the problem presented here...  The basic task in 
> > > scsi_tgt_if's case is just shoveling packets to/from userspace.
> > ..
> > 
> > Except it's rather SCSI specific, and the userspace frontend even more so.
> > The code expects a SCSI command block, LUN, TAG, and other fields that
> > a SATA FIS won't have.  Seems clumsy, particularly when we (in theory)
> > are trying to decouple libata from SCSI.   But if that's the way,
> > then I can clumsily wrap each FIS in a fake ATA_16 header or something.
> 
> Yes, but SG_IO began life as being entirely SCSI specific, but it's now
> the block packet infrastructure.  I don't think anyone has an objection
> to the SCSI target infrastructure becoming block target
> infrastructure ... the fields can be made optional or more generic to
> further this.  Probably it should speak a generic packet with a protocol
> identifier and let the user space sort out the details.

Mike Christie and I designed a generic target framework, not SCSI
specific. But we decided to make the framework SCSI specific because
we thought that nobody except for SCSI wanted the target
framework. Seems that we were wrong. Let me now if I can help make a
generic target framework out of the SCSI target framework.

Re: SATA "target mode" (or "Channel-to-Channel" comm mode)

[Jeff Garzik]

Oh BTW...  as a reminder, there is an AHCI clone from ATP out there, 
that also supports SATA target mode.

They didn't post a URL to docs, as I recall, but they did make the docs 
freely available to any who asked.  I have a copy and can email, if 
anyone wants to take a look.

Also FWIW, the AHCI driver needed a few modifications (that never got 
done) to support the ATP variant of AHCI.

	Jeff