PMP and SEMB messages to SEP

PMP and SEMB messages to SEP

[Herbert Poetzl]

I'm currently trying to wrap my head around SEMB and sending
messages to an attached SEP on some Port Multiplier connected
via eSATA ...

I found that the AHCI driver supports certain well defined
messages (for the activity leds) but I'm not sure what they
actually are, and I have no idea how to utilize them from
userspace at the moment

Basically what I want to do is the following:

 - attach a Smart Enclosure via I2C to the PMP
   (the SEP is basically under my control)
 - retrieve certain values like temperature readings
   from that Smart Enclosure via the SEMB
 - send specific 'command' messages to the SEP to
   control user visible lights on the enclosure

My current Test Setup includes a simple Sil 3726 based PMP
(Dawicontrol DC-6510 PM) which has a connector for a SEP 
using I2C (via the SEMB), connected to an eSATA Port which 
is PMP capable and handles the connected drives quite well.

I had a quick read on the (S)ATA specification regarding
PMP and sending/retrieving messages from the SEMB, but at
the moment, I have no clue how to 'map' that to existing? 
code in the Linux kernel or how that should/could be exposed 
to userspace ...

Any hints/pointers/information would be greatly appreciated.

Many thanks in advance,
Herbert 

Re: PMP and SEMB messages to SEP

[Tejun Heo]

Hello,

On Thu, Jan 13, 2011 at 08:25:58PM +0100, Herbert Poetzl wrote:
> I had a quick read on the (S)ATA specification regarding
> PMP and sending/retrieving messages from the SEMB, but at
> the moment, I have no clue how to 'map' that to existing? 
> code in the Linux kernel or how that should/could be exposed 
> to userspace ...
> 
> Any hints/pointers/information would be greatly appreciated.

Unfortunately, it hasn't been implemented yet.  I didn't have access
to any SEMB aware device and no one else worked on it, so...

-- 
tejun

Re: PMP and SEMB messages to SEP

[Herbert Poetzl]

On Fri, Jan 14, 2011 at 03:20:18PM +0100, Tejun Heo wrote:
> Hello,

> On Thu, Jan 13, 2011 at 08:25:58PM +0100, Herbert Poetzl wrote:
> > I had a quick read on the (S)ATA specification regarding
> > PMP and sending/retrieving messages from the SEMB, but at
> > the moment, I have no clue how to 'map' that to existing? 
> > code in the Linux kernel or how that should/could be exposed 
> > to userspace ...

> > Any hints/pointers/information would be greatly appreciated.

> Unfortunately, it hasn't been implemented yet.  

Okay, so any ideas/suggestions how to implement it?

> I didn't have access to any SEMB aware device and no one 
> else worked on it, so...

No problem there, AFAIK, any SiI3726 based PMP should have
everything required for a SEMB, but most PMP products do not
add the required header to actually connect it to a SEP
(at lest the SiI3726 specification says so :)

I have no problem digging into it and doing the testing, but
of course, all input on how to address this is quite welcome

Thanks in advance,
Herbert

> -- 
> tejun

Re: PMP and SEMB messages to SEP

[Tejun Heo]

Hello,

On Fri, Jan 14, 2011 at 05:59:57PM +0100, Herbert Poetzl wrote:
> > Unfortunately, it hasn't been implemented yet.  
> 
> Okay, so any ideas/suggestions how to implement it?

I'll be happy to answer questions / review patches but I don't really
have much idea how it should be implemented at this point.  I guess
libata would just serve as transport layer for the enclosure
management high level driver (there were two variants, IIRC).  There
also was something added to ahci which is exported through sysfs.

Good luck!

-- 
tejun

Re: PMP and SEMB messages to SEP

[Herbert Poetzl]

On Fri, Jan 14, 2011 at 06:04:07PM +0100, Tejun Heo wrote:
> Hello,

> On Fri, Jan 14, 2011 at 05:59:57PM +0100, Herbert Poetzl wrote:
> > > Unfortunately, it hasn't been implemented yet.  

> > Okay, so any ideas/suggestions how to implement it?

> I'll be happy to answer questions / review patches but I don't
> really have much idea how it should be implemented at this point. 
> I guess libata would just serve as transport layer for the 
> enclosure management high level driver (there were two variants, 
> IIRC). 

Fair enough, let's start with a bunch of simple questions
I got from looking at the code (which I did just recently,
so don't expect deep understanding of the inner workings,
or even intelligent questions FWIW :)

- libata is used in the kernel for ata, sata and to some
  extend for scsi (via (s)ata), as a helper library.
  is there any counterpart in userspace to communicate
  with (s)ata host/devices or how are the interfaces to
  userspace designed to work?

- the SiI3726 supports SAF-TE and SES protocols for the
  SEMB/SEP and the EMCs are sent through the SATA interface
  (according to the docu, SEP_ATTN in the command register
  and SEP command code in the features register)

  + how would I send such a command and retrieve the result
    (if there is one) from within the kernel (maybe with
    libata or from the ahci layer)?

- the SiI3726 supports GPIO pins, which can be reached via
  the General Status and Control Register [130] and accoring
  to the docu, the Read/Write Port Multiplier command can
  be used to read/write that register.
 
  + how would I go about issuing such a command and where
    should it be done? i.e. what about interference with
    other commands? what about retrieving return values?

> There also was something added to ahci which is exported through
> sysfs.

I saw that the ahci driver uses *em_message* which I
think might be related to sending activity messages and
it also lists capabilities (which include 'ems') on
host adapters, but I'm not entirely sure this is SEMB
related (yet)

thanks for your help,
Herbert

> Good luck!
> 
> -- 
> tejun

Re: PMP and SEMB messages to SEP

[Tejun Heo]

(cc'ing linux-scsi)
Hello,

On Fri, Jan 14, 2011 at 06:37:59PM +0100, Herbert Poetzl wrote:
> Fair enough, let's start with a bunch of simple questions
> I got from looking at the code (which I did just recently,
> so don't expect deep understanding of the inner workings,
> or even intelligent questions FWIW :)
> 
> - libata is used in the kernel for ata, sata and to some
>   extend for scsi (via (s)ata), as a helper library.

The (s) there is Serial not SCSI, but yes some SAS drivers use libata
to drive SATA devices attached to SAS controllers.

>   is there any counterpart in userspace to communicate
>   with (s)ata host/devices or how are the interfaces to
>   userspace designed to work?

SG_IO is the standard way to issue custom commands.  You need to wrap
ATA command inside a SCSI SCSI-ATA passthrough command.  Take a look
at hdparm, smartctl or any other tool which issues direct commands.

> - the SiI3726 supports SAF-TE and SES protocols for the
>   SEMB/SEP and the EMCs are sent through the SATA interface
>   (according to the docu, SEP_ATTN in the command register
>   and SEP command code in the features register)
> 
>   + how would I send such a command and retrieve the result
>     (if there is one) from within the kernel (maybe with
>     libata or from the ahci layer)?

If it's a proper command SAT passthrough via SG_IO should work but I'm
not sure whether it is.  Is it?

> - the SiI3726 supports GPIO pins, which can be reached via
>   the General Status and Control Register [130] and accoring
>   to the docu, the Read/Write Port Multiplier command can
>   be used to read/write that register.
>  
>   + how would I go about issuing such a command and where
>     should it be done? i.e. what about interference with
>     other commands? what about retrieving return values?

The problem is that the PMP device itself is currently not allocated a
userland visible device, so it doesn't have any /dev/* node.  Hmmm...

> > There also was something added to ahci which is exported through
> > sysfs.
> 
> I saw that the ahci driver uses *em_message* which I
> think might be related to sending activity messages and
> it also lists capabilities (which include 'ems') on
> host adapters, but I'm not entirely sure this is SEMB
> related (yet)

The message sent there can be of any format and that's why it was
added as a separate sysfs node.

ISTR there's a SCSI enclosure management driver.  Maybe SCSI people
have better clues about how enclosure management should be done?

Thanks.

-- 
tejun

Re: PMP and SEMB messages to SEP

[James Bottomley]

On Mon, 2011-01-17 at 16:40 +0100, Tejun Heo wrote:
> (cc'ing linux-scsi)
> Hello,
> 
> On Fri, Jan 14, 2011 at 06:37:59PM +0100, Herbert Poetzl wrote:
> > Fair enough, let's start with a bunch of simple questions
> > I got from looking at the code (which I did just recently,
> > so don't expect deep understanding of the inner workings,
> > or even intelligent questions FWIW :)
> > 
> > - libata is used in the kernel for ata, sata and to some
> >   extend for scsi (via (s)ata), as a helper library.
> 
> The (s) there is Serial not SCSI, but yes some SAS drivers use libata
> to drive SATA devices attached to SAS controllers.
> 
> >   is there any counterpart in userspace to communicate
> >   with (s)ata host/devices or how are the interfaces to
> >   userspace designed to work?
> 
> SG_IO is the standard way to issue custom commands.  You need to wrap
> ATA command inside a SCSI SCSI-ATA passthrough command.  Take a look
> at hdparm, smartctl or any other tool which issues direct commands.
> 
> > - the SiI3726 supports SAF-TE and SES protocols for the
> >   SEMB/SEP and the EMCs are sent through the SATA interface
> >   (according to the docu, SEP_ATTN in the command register
> >   and SEP command code in the features register)
> > 
> >   + how would I send such a command and retrieve the result
> >     (if there is one) from within the kernel (maybe with
> >     libata or from the ahci layer)?
> 
> If it's a proper command SAT passthrough via SG_IO should work but I'm
> not sure whether it is.  Is it?

it sounds like it can be sent via ATA_12 or ATA_16 then depending on
what it does to the device state model.

> > - the SiI3726 supports GPIO pins, which can be reached via
> >   the General Status and Control Register [130] and accoring
> >   to the docu, the Read/Write Port Multiplier command can
> >   be used to read/write that register.
> >  
> >   + how would I go about issuing such a command and where
> >     should it be done? i.e. what about interference with
> >     other commands? what about retrieving return values?
> 
> The problem is that the PMP device itself is currently not allocated a
> userland visible device, so it doesn't have any /dev/* node.  Hmmm...

So perhaps it should be.  If you look at the equivalent topology on SAS,
our expanders have a bsg device node precisely so that we can do this.

That said, SAS expanders have a defined protocol (SAS Management
Protocol) to talk to the outside world, so they are real visible objects
always in our topology ... I'm not sure PMP has this ... it seems that
all PMP visibility is an extension to the standard?

> > > There also was something added to ahci which is exported through
> > > sysfs.
> > 
> > I saw that the ahci driver uses *em_message* which I
> > think might be related to sending activity messages and
> > it also lists capabilities (which include 'ems') on
> > host adapters, but I'm not entirely sure this is SEMB
> > related (yet)
> 
> The message sent there can be of any format and that's why it was
> added as a separate sysfs node.
> 
> ISTR there's a SCSI enclosure management driver.  Maybe SCSI people
> have better clues about how enclosure management should be done?

Sure, as long as it speaks standard ses-2, there shouldn't be a protocol
problem.  The main problem is recognition: ses has to bind to an
enclosure device.  It can bind either to an explicit device (about all
the enclosures I've seen so far) where the ses device has a separate
address in the SCSI topology or an implicit device (where another SCSI
device indicates it has an enclosure port embedded in it).  As currently
coded, our ses driver only does the former probably the best way is to
expose the ses device via libata and we'll simply bind to it.

James


However, 

Re: PMP and SEMB messages to SEP

[Herbert Poetzl]

On Mon, Jan 17, 2011 at 10:18:51AM -0600, James Bottomley wrote:
> On Mon, 2011-01-17 at 16:40 +0100, Tejun Heo wrote:
> > (cc'ing linux-scsi)
> > Hello,
> > 
> > On Fri, Jan 14, 2011 at 06:37:59PM +0100, Herbert Poetzl wrote:
> > > Fair enough, let's start with a bunch of simple questions
> > > I got from looking at the code (which I did just recently,
> > > so don't expect deep understanding of the inner workings,
> > > or even intelligent questions FWIW :)
> > > 
> > > - libata is used in the kernel for ata, sata and to some
> > >   extend for scsi (via (s)ata), as a helper library.
> > 
> > The (s) there is Serial not SCSI, but yes some SAS drivers use libata
> > to drive SATA devices attached to SAS controllers.
> > 
> > >   is there any counterpart in userspace to communicate
> > >   with (s)ata host/devices or how are the interfaces to
> > >   userspace designed to work?
> > 
> > SG_IO is the standard way to issue custom commands.  You need to wrap
> > ATA command inside a SCSI SCSI-ATA passthrough command.  Take a look
> > at hdparm, smartctl or any other tool which issues direct commands.
> > 
> > > - the SiI3726 supports SAF-TE and SES protocols for the
> > >   SEMB/SEP and the EMCs are sent through the SATA interface
> > >   (according to the docu, SEP_ATTN in the command register
> > >   and SEP command code in the features register)
> > > 
> > >   + how would I send such a command and retrieve the result
> > >     (if there is one) from within the kernel (maybe with
> > >     libata or from the ahci layer)?
> > 
> > If it's a proper command SAT passthrough via SG_IO should work but I'm
> > not sure whether it is.  Is it?
> 
> it sounds like it can be sent via ATA_12 or ATA_16 then depending on
> what it does to the device state model.
> 
> > > - the SiI3726 supports GPIO pins, which can be reached via
> > >   the General Status and Control Register [130] and accoring
> > >   to the docu, the Read/Write Port Multiplier command can
> > >   be used to read/write that register.
> > >  
> > >   + how would I go about issuing such a command and where
> > >     should it be done? i.e. what about interference with
> > >     other commands? what about retrieving return values?
> > 
> > The problem is that the PMP device itself is currently not allocated
> > a userland visible device, so it doesn't have any /dev/* node.
> > Hmmm...
>
> So perhaps it should be. If you look at the equivalent topology on
> SAS, our expanders have a bsg device node precisely so that we can do
> this.

I agree, the PMP should get a device or at least some kind of
interface to address them, especially as the SATA topology can
get quite complicated too, for example it seems that PMPs can
be daisy chained in an infinite sequence like this

       /      /      /
 --[PM]---[PM]---[PM]-- - -
       \      \      \

> That said, SAS expanders have a defined protocol (SAS Management
> Protocol) to talk to the outside world, so they are real visible objects
> always in our topology ... I'm not sure PMP has this ... it seems that
> all PMP visibility is an extension to the standard?

At least to me it seems, PMPs do have a defined protocol as
well, otherwise the PM 1.0/1.1 protocol specificiation would
not make much sense, IMHO ...

> > > > There also was something added to ahci which is exported through
> > > > sysfs.
> > > 
> > > I saw that the ahci driver uses *em_message* which I
> > > think might be related to sending activity messages and
> > > it also lists capabilities (which include 'ems') on
> > > host adapters, but I'm not entirely sure this is SEMB
> > > related (yet)
> > 
> > The message sent there can be of any format and that's why it was
> > added as a separate sysfs node.
> > 
> > ISTR there's a SCSI enclosure management driver.  Maybe SCSI people
> > have better clues about how enclosure management should be done?
> 
> Sure, as long as it speaks standard ses-2, there shouldn't be a
> protocol problem. The main problem is recognition: ses has to bind
> to an enclosure device. It can bind either to an explicit device
> (about all the enclosures I've seen so far) where the ses device has
> a separate address in the SCSI topology or an implicit device (where
> another SCSI device indicates it has an enclosure port embedded in
> it). As currently coded, our ses driver only does the former probably
> the best way is to expose the ses device via libata and we'll simply
> bind to it.

so AHCI em_messages use standard ses-2 or did I misinterpret
this (for me cryptical) information?

in any case, thanks,
Herbert

> James
> 
> However, 

Re: PMP and SEMB messages to SEP

[Tejun Heo]

On Mon, Jan 17, 2011 at 06:13:12PM +0100, Herbert Poetzl wrote:
> > So perhaps it should be. If you look at the equivalent topology on
> > SAS, our expanders have a bsg device node precisely so that we can do
> > this.
> 
> I agree, the PMP should get a device or at least some kind of
> interface to address them, especially as the SATA topology can
> get quite complicated too, for example it seems that PMPs can
> be daisy chained in an infinite sequence like this
> 
>        /      /      /
>  --[PM]---[PM]---[PM]-- - -
>        \      \      \

Oh, no, that's not allowed.  You can't address destinations that way.
PMP is primarily a switch, not a router.

> > Sure, as long as it speaks standard ses-2, there shouldn't be a
> > protocol problem. The main problem is recognition: ses has to bind
> > to an enclosure device. It can bind either to an explicit device
> > (about all the enclosures I've seen so far) where the ses device has
> > a separate address in the SCSI topology or an implicit device (where
> > another SCSI device indicates it has an enclosure port embedded in
> > it). As currently coded, our ses driver only does the former probably
> > the best way is to expose the ses device via libata and we'll simply
> > bind to it.
> 
> so AHCI em_messages use standard ses-2 or did I misinterpret
> this (for me cryptical) information?

AFAIK, it just doesn't care.  It could be ses-2 or whatever else.  It
just transmits the binary blob it receives via sysfs and vice-versa.

Thanks.

-- 
tejun

Re: PMP and SEMB messages to SEP

[Herbert Poetzl]

On Mon, Jan 17, 2011 at 06:20:54PM +0100, Tejun Heo wrote:
> On Mon, Jan 17, 2011 at 06:13:12PM +0100, Herbert Poetzl wrote:
> > > So perhaps it should be. If you look at the equivalent topology on
> > > SAS, our expanders have a bsg device node precisely so that we can do
> > > this.
> > 
> > I agree, the PMP should get a device or at least some kind of
> > interface to address them, especially as the SATA topology can
> > get quite complicated too, for example it seems that PMPs can
> > be daisy chained in an infinite sequence like this
> > 
> >        /      /      /
> >  --[PM]---[PM]---[PM]-- - -
> >        \      \      \
> 
> Oh, no, that's not allowed.  You can't address destinations that way.
> PMP is primarily a switch, not a router.

hmm, haven't had the time to dig through the specs yet, but
at least addonics seems to disagree here:

http://www.addonics.com/products/host_controller/tutorial_pm.asp

but maybe that's just a marketing page with no relation
to reality ....

> > > Sure, as long as it speaks standard ses-2, there shouldn't be a
> > > protocol problem. The main problem is recognition: ses has to bind
> > > to an enclosure device. It can bind either to an explicit device
> > > (about all the enclosures I've seen so far) where the ses device has
> > > a separate address in the SCSI topology or an implicit device (where
> > > another SCSI device indicates it has an enclosure port embedded in
> > > it). As currently coded, our ses driver only does the former probably
> > > the best way is to expose the ses device via libata and we'll simply
> > > bind to it.
> > 
> > so AHCI em_messages use standard ses-2 or did I misinterpret
> > this (for me cryptical) information?
> 
> AFAIK, it just doesn't care.  It could be ses-2 or whatever else.  It
> just transmits the binary blob it receives via sysfs and vice-versa.

the interesting part here is, that the AHCI host controller
my PMP is connected to recognizes the PMP perfectly fine
(i.e. more than one drive works just as expected), but doesn't
seem to allow for the em_message part (despite the fact that
the attached PMP seems to be SEMB/SEP capable) ...

maybe this is just a bug in the kernel code, maybe the AHCI
implementation doesn't allow a PMP to receive enclosure
management messages at all ...

best,
Herbert

> Thanks.
> 
> -- 
> tejun

Re: PMP and SEMB messages to SEP

[Tejun Heo]

On Mon, Jan 17, 2011 at 06:34:33PM +0100, Herbert Poetzl wrote:
> hmm, haven't had the time to dig through the specs yet, but
> at least addonics seems to disagree here:
> 
> http://www.addonics.com/products/host_controller/tutorial_pm.asp
> 
> but maybe that's just a marketing page with no relation
> to reality ....

That's hardware RAID PMP.  It would just show up as a single drive to
the host.  They can do whatever they want to if they do that.

> > AFAIK, it just doesn't care.  It could be ses-2 or whatever else.  It
> > just transmits the binary blob it receives via sysfs and vice-versa.
> 
> the interesting part here is, that the AHCI host controller
> my PMP is connected to recognizes the PMP perfectly fine
> (i.e. more than one drive works just as expected), but doesn't
> seem to allow for the em_message part (despite the fact that
> the attached PMP seems to be SEMB/SEP capable) ...
> 
> maybe this is just a bug in the kernel code, maybe the AHCI
> implementation doesn't allow a PMP to receive enclosure
> management messages at all ...

AFAIK, the ahci em message thing is not via PMP.  It's for cases where
the ahci controller is directly connected to an enclosure.  Well,
that's my understanding anyway.

-- 
tejun

Re: PMP and SEMB messages to SEP

[Herbert Poetzl]

On Mon, Jan 17, 2011 at 06:39:46PM +0100, Tejun Heo wrote:
> On Mon, Jan 17, 2011 at 06:34:33PM +0100, Herbert Poetzl wrote:
> > hmm, haven't had the time to dig through the specs yet, but
> > at least addonics seems to disagree here:
> > 
> > http://www.addonics.com/products/host_controller/tutorial_pm.asp
> > 
> > but maybe that's just a marketing page with no relation
> > to reality ....
> 
> That's hardware RAID PMP.  It would just show up as a single drive to
> the host.  They can do whatever they want to if they do that.

okay, understood.

> > > AFAIK, it just doesn't care.  It could be ses-2 or whatever else.  It
> > > just transmits the binary blob it receives via sysfs and vice-versa.
> > 
> > the interesting part here is, that the AHCI host controller
> > my PMP is connected to recognizes the PMP perfectly fine
> > (i.e. more than one drive works just as expected), but doesn't
> > seem to allow for the em_message part (despite the fact that
> > the attached PMP seems to be SEMB/SEP capable) ...
> > 
> > maybe this is just a bug in the kernel code, maybe the AHCI
> > implementation doesn't allow a PMP to receive enclosure
> > management messages at all ...
> 
> AFAIK, the ahci em message thing is not via PMP. 
> It's for cases where the ahci controller is directly connected 
> to an enclosure. Well, that's my understanding anyway.

okay, so something similar would be useful for controlling
enclosures attached to PMPs then?

what would be the 'proper' place in the sysfs tree, espceically
as the PMP doesn't show up there (yet)?

how would I go about adding something like that to the kernel
and where should it be placed (sysfs and code wise)?

thanks in advance,
Herbert

> -- 
> tejun

Re: PMP and SEMB messages to SEP

[James Bottomley]

On Mon, 2011-01-17 at 18:48 +0100, Herbert Poetzl wrote:
> On Mon, Jan 17, 2011 at 06:39:46PM +0100, Tejun Heo wrote:
> > On Mon, Jan 17, 2011 at 06:34:33PM +0100, Herbert Poetzl wrote:
> > > hmm, haven't had the time to dig through the specs yet, but
> > > at least addonics seems to disagree here:
> > > 
> > > http://www.addonics.com/products/host_controller/tutorial_pm.asp
> > > 
> > > but maybe that's just a marketing page with no relation
> > > to reality ....
> > 
> > That's hardware RAID PMP.  It would just show up as a single drive to
> > the host.  They can do whatever they want to if they do that.
> 
> okay, understood.
> 
> > > > AFAIK, it just doesn't care.  It could be ses-2 or whatever else.  It
> > > > just transmits the binary blob it receives via sysfs and vice-versa.
> > > 
> > > the interesting part here is, that the AHCI host controller
> > > my PMP is connected to recognizes the PMP perfectly fine
> > > (i.e. more than one drive works just as expected), but doesn't
> > > seem to allow for the em_message part (despite the fact that
> > > the attached PMP seems to be SEMB/SEP capable) ...
> > > 
> > > maybe this is just a bug in the kernel code, maybe the AHCI
> > > implementation doesn't allow a PMP to receive enclosure
> > > management messages at all ...
> > 
> > AFAIK, the ahci em message thing is not via PMP. 
> > It's for cases where the ahci controller is directly connected 
> > to an enclosure. Well, that's my understanding anyway.
> 
> okay, so something similar would be useful for controlling
> enclosures attached to PMPs then?
> 
> what would be the 'proper' place in the sysfs tree, espceically
> as the PMP doesn't show up there (yet)?
> 
> how would I go about adding something like that to the kernel
> and where should it be placed (sysfs and code wise)?

OK, just so we're not at cross purposes:

     1. For an enclosure processor to show up and be attached to the
        enclosure driver, it has to present as a SCSI target.  This
        often means handling nasty encapsulations somewhere (SES is most
        often used on non-standard busses like i2c or GPIO).  If it's
        within a PMP and has a defined encapsulation which ATA
        recognises, it sounds like libata is the best place for this.
     2. The above has nothing to do with whether the PMP device shows up
        in sysfs or has a bsg command port.  That's a completely
        separate discussion based on whether it makes sense (and whether
        we can cope with the topological complexity).  Just by way of
        example, it's mostly not possible to use expanders with
        enclosure devices via our expander bsg device, because there's
        no SMP encapsulation of SES.  What mostly happens is that the
        expander has two or more target ports, one of which is SMP and
        the other (completely separate one) is an addressable SES
        target.

James

Re: PMP and SEMB messages to SEP

[Herbert Poetzl]

On Mon, Jan 17, 2011 at 12:05:04PM -0600, James Bottomley wrote:
> On Mon, 2011-01-17 at 18:48 +0100, Herbert Poetzl wrote:
> > On Mon, Jan 17, 2011 at 06:39:46PM +0100, Tejun Heo wrote:
> > > On Mon, Jan 17, 2011 at 06:34:33PM +0100, Herbert Poetzl wrote:
> > > > hmm, haven't had the time to dig through the specs yet, but
> > > > at least addonics seems to disagree here:
> > > > 
> > > > http://www.addonics.com/products/host_controller/tutorial_pm.asp
> > > > 
> > > > but maybe that's just a marketing page with no relation
> > > > to reality ....
> > > 
> > > That's hardware RAID PMP.  It would just show up as a single drive to
> > > the host.  They can do whatever they want to if they do that.
> > 
> > okay, understood.
> > 
> > > > > AFAIK, it just doesn't care.  It could be ses-2 or whatever else.  It
> > > > > just transmits the binary blob it receives via sysfs and vice-versa.
> > > > 
> > > > the interesting part here is, that the AHCI host controller
> > > > my PMP is connected to recognizes the PMP perfectly fine
> > > > (i.e. more than one drive works just as expected), but doesn't
> > > > seem to allow for the em_message part (despite the fact that
> > > > the attached PMP seems to be SEMB/SEP capable) ...
> > > > 
> > > > maybe this is just a bug in the kernel code, maybe the AHCI
> > > > implementation doesn't allow a PMP to receive enclosure
> > > > management messages at all ...
> > > 
> > > AFAIK, the ahci em message thing is not via PMP. 
> > > It's for cases where the ahci controller is directly connected 
> > > to an enclosure. Well, that's my understanding anyway.
> > 
> > okay, so something similar would be useful for controlling
> > enclosures attached to PMPs then?
> > 
> > what would be the 'proper' place in the sysfs tree, espceically
> > as the PMP doesn't show up there (yet)?
> > 
> > how would I go about adding something like that to the kernel
> > and where should it be placed (sysfs and code wise)?
> 
> OK, just so we're not at cross purposes:
> 
>      1. For an enclosure processor to show up and be attached to the
>         enclosure driver, it has to present as a SCSI target. This
>         often means handling nasty encapsulations somewhere (SES is
>         most often used on non-standard busses like i2c or GPIO). If
>         it's within a PMP and has a defined encapsulation which ATA
>         recognises, it sounds like libata is the best place for this.

it seems that the SATA specification, specifically the PMP
part 'defines' something called SEMB (Smart Enclosure Management
Bridge) which is controlled via specific SATA PMP commands to
send 'messages' over (typically) I2C to an SEP (Smart Enclosure
Processor) which can act upon them and even generate data/replies
to retrieve information like the temperature, fan status, etc

>      2. The above has nothing to do with whether the PMP device shows 
>         up in sysfs or has a bsg command port. That's a completely
>         separate discussion based on whether it makes sense (and
>         whether we can cope with the topological complexity). Just by
>         way of example, it's mostly not possible to use expanders with
>         enclosure devices via our expander bsg device, because there's
>         no SMP encapsulation of SES. What mostly happens is that the
>         expander has two or more target ports, one of which is SMP
>         and the other (completely separate one) is an addressable SES
>         target.

as far as I understood, the SEMB is a separate and well
defined entity controllable via special PMP commands ...

not every PMP has to have one, but if they do, it should
conform to the specification, even on the I2C layer

HTC,
Herbert

> James
> 

Re: PMP and SEMB messages to SEP

[Herbert Poetzl]

On Mon, Jan 17, 2011 at 07:21:40PM +0100, Herbert Poetzl wrote:
> On Mon, Jan 17, 2011 at 12:05:04PM -0600, James Bottomley wrote:
> > On Mon, 2011-01-17 at 18:48 +0100, Herbert Poetzl wrote:
> > > On Mon, Jan 17, 2011 at 06:39:46PM +0100, Tejun Heo wrote:
> > > > On Mon, Jan 17, 2011 at 06:34:33PM +0100, Herbert Poetzl wrote:
> > > > > hmm, haven't had the time to dig through the specs yet, but
> > > > > at least addonics seems to disagree here:
> > > > > 
> > > > > http://www.addonics.com/products/host_controller/tutorial_pm.asp
> > > > > 
> > > > > but maybe that's just a marketing page with no relation
> > > > > to reality ....
> > > > 
> > > > That's hardware RAID PMP.  It would just show up as a single drive to
> > > > the host.  They can do whatever they want to if they do that.
> > > 
> > > okay, understood.
> > > 
> > > > > > AFAIK, it just doesn't care.  It could be ses-2 or whatever else.  It
> > > > > > just transmits the binary blob it receives via sysfs and vice-versa.
> > > > > 
> > > > > the interesting part here is, that the AHCI host controller
> > > > > my PMP is connected to recognizes the PMP perfectly fine
> > > > > (i.e. more than one drive works just as expected), but doesn't
> > > > > seem to allow for the em_message part (despite the fact that
> > > > > the attached PMP seems to be SEMB/SEP capable) ...
> > > > > 
> > > > > maybe this is just a bug in the kernel code, maybe the AHCI
> > > > > implementation doesn't allow a PMP to receive enclosure
> > > > > management messages at all ...
> > > > 
> > > > AFAIK, the ahci em message thing is not via PMP. 
> > > > It's for cases where the ahci controller is directly connected 
> > > > to an enclosure. Well, that's my understanding anyway.
> > > 
> > > okay, so something similar would be useful for controlling
> > > enclosures attached to PMPs then?
> > > 
> > > what would be the 'proper' place in the sysfs tree, espceically
> > > as the PMP doesn't show up there (yet)?
> > > 
> > > how would I go about adding something like that to the kernel
> > > and where should it be placed (sysfs and code wise)?
> > 
> > OK, just so we're not at cross purposes:
> > 
> >      1. For an enclosure processor to show up and be attached to the
> >         enclosure driver, it has to present as a SCSI target. This
> >         often means handling nasty encapsulations somewhere (SES is
> >         most often used on non-standard busses like i2c or GPIO). If
> >         it's within a PMP and has a defined encapsulation which ATA
> >         recognises, it sounds like libata is the best place for this.
> 
> it seems that the SATA specification, specifically the PMP
> part 'defines' something called SEMB (Smart Enclosure Management
> Bridge) which is controlled via specific SATA PMP commands to
> send 'messages' over (typically) I2C to an SEP (Smart Enclosure
> Processor) which can act upon them and even generate data/replies
> to retrieve information like the temperature, fan status, etc

actually SEMS stands for Seria ATA Enclosure Management Bridge,
and SEP for Storage Enclosure Processor .. just looked up the
terms in the specification ...

> >      2. The above has nothing to do with whether the PMP device shows 
> >         up in sysfs or has a bsg command port. That's a completely
> >         separate discussion based on whether it makes sense (and
> >         whether we can cope with the topological complexity). Just by
> >         way of example, it's mostly not possible to use expanders with
> >         enclosure devices via our expander bsg device, because there's
> >         no SMP encapsulation of SES. What mostly happens is that the
> >         expander has two or more target ports, one of which is SMP
> >         and the other (completely separate one) is an addressable SES
> >         target.

> as far as I understood, the SEMB is a separate and well
> defined entity controllable via special PMP commands ...

> not every PMP has to have one, but if they do, it should
> conform to the specification, even on the I2C layer

the protocols used on I2C is IPMI, the command protocol to
communicate with the SEP is either SAF-TE or SES, AFAICT
but it might be proprietary if both the SEMB and the SEP
are from the same manufacturer ...

sorry for the confusion,
Herbert

> HTC,
> Herbert
> 
> > James
> > 

Re: PMP and SEMB messages to SEP

[Tejun Heo]

Hello, James.

On Mon, Jan 17, 2011 at 10:18:51AM -0600, James Bottomley wrote:
> > The problem is that the PMP device itself is currently not allocated a
> > userland visible device, so it doesn't have any /dev/* node.  Hmmm...
> 
> So perhaps it should be.  If you look at the equivalent topology on SAS,
> our expanders have a bsg device node precisely so that we can do this.
>
> That said, SAS expanders have a defined protocol (SAS Management
> Protocol) to talk to the outside world, so they are real visible objects
> always in our topology ... I'm not sure PMP has this ... it seems that
> all PMP visibility is an extension to the standard?

SATA PMP is mostly a dumb switch and there isn't much which can be
done by issuing custom commands (and IIRC we didn't have bsg back
then), so it was never made visible to userland, but yeah probably
exporting a bsg node is a good idea.  Inside libata, the PMP device
has its device representation and all so it shouldn't be too difficult
either.  Not really sure how the inquiry and stuff should be handled
tho.

Thanks.

-- 
tejun

Re: PMP and SEMB messages to SEP

[James Bottomley]

On Mon, 2011-01-17 at 18:18 +0100, Tejun Heo wrote:
> Hello, James.
> 
> On Mon, Jan 17, 2011 at 10:18:51AM -0600, James Bottomley wrote:
> > > The problem is that the PMP device itself is currently not allocated a
> > > userland visible device, so it doesn't have any /dev/* node.  Hmmm...
> > 
> > So perhaps it should be.  If you look at the equivalent topology on SAS,
> > our expanders have a bsg device node precisely so that we can do this.
> >
> > That said, SAS expanders have a defined protocol (SAS Management
> > Protocol) to talk to the outside world, so they are real visible objects
> > always in our topology ... I'm not sure PMP has this ... it seems that
> > all PMP visibility is an extension to the standard?
> 
> SATA PMP is mostly a dumb switch and there isn't much which can be
> done by issuing custom commands (and IIRC we didn't have bsg back
> then), so it was never made visible to userland, but yeah probably
> exporting a bsg node is a good idea.  Inside libata, the PMP device
> has its device representation and all so it shouldn't be too difficult
> either.  Not really sure how the inquiry and stuff should be handled
> tho.

Expanders aren't SCSI devices either ... that means they don't appear as
visible to standard SCSI mechanisms like INQUIRY (SMP isn't a SCSI
protocol it's a SAS extension).  They just appear as part of the
topology in the device tree.

James

Re: PMP and SEMB messages to SEP

[Tejun Heo]

On Mon, Jan 17, 2011 at 11:21:46AM -0600, James Bottomley wrote:
> Expanders aren't SCSI devices either ... that means they don't appear as
> visible to standard SCSI mechanisms like INQUIRY (SMP isn't a SCSI
> protocol it's a SAS extension).  They just appear as part of the
> topology in the device tree.

Ah, okay.  Then, things would be much simpler.  I was worrying about
how it would map to SCSI layer including INQUIRY emulations and all
those stuff.  Thanks for the info.

-- 
tejun

Re: PMP and SEMB messages to SEP

[Herbert Poetzl]

On Mon, Jan 17, 2011 at 06:24:15PM +0100, Tejun Heo wrote:
> On Mon, Jan 17, 2011 at 11:21:46AM -0600, James Bottomley wrote:
> > Expanders aren't SCSI devices either ... that means they don't
> > appear as visible to standard SCSI mechanisms like INQUIRY (SMP
> > isn't a SCSI protocol it's a SAS extension). They just appear as
> > part of the topology in the device tree.

> Ah, okay.  Then, things would be much simpler.  I was worrying about
> how it would map to SCSI layer including INQUIRY emulations and all
> those stuff.  Thanks for the info.

So, any thoughts on how and where to add the PMP to the devices
available for sending SCSI or ATA commands to?

I'm willing to try to add the required code, but I'm sure I'll
benefit from a few hints from the experts ...

thanks,
Herbert

> -- 
> tejun
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ide" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: PMP and SEMB messages to SEP

[James Bottomley]

On Tue, 2011-01-25 at 04:07 +0100, Herbert Poetzl wrote:
> On Mon, Jan 17, 2011 at 06:24:15PM +0100, Tejun Heo wrote:
> > On Mon, Jan 17, 2011 at 11:21:46AM -0600, James Bottomley wrote:
> > > Expanders aren't SCSI devices either ... that means they don't
> > > appear as visible to standard SCSI mechanisms like INQUIRY (SMP
> > > isn't a SCSI protocol it's a SAS extension). They just appear as
> > > part of the topology in the device tree.
> 
> > Ah, okay.  Then, things would be much simpler.  I was worrying about
> > how it would map to SCSI layer including INQUIRY emulations and all
> > those stuff.  Thanks for the info.
> 
> So, any thoughts on how and where to add the PMP to the devices
> available for sending SCSI or ATA commands to?
> 
> I'm willing to try to add the required code, but I'm sure I'll
> benefit from a few hints from the experts ...

Well, I'm not sure I'd count as an expert on this piece:  I haven't read
the relevant ATA standards.  From what I understand, SES packets are
encapsulated over an ATA command for SEMB.  In that case, it should be
fairly simple to recognise this and present a SCSI device which simply
encapsulates everything sent to it over this protocol.  That would allow
the ses ULD to attach seamlessly.

James

Re: PMP and SEMB messages to SEP

[Herbert Poetzl]

On Tue, Jan 25, 2011 at 09:14:14AM -0500, James Bottomley wrote:
> On Tue, 2011-01-25 at 04:07 +0100, Herbert Poetzl wrote:
>> On Mon, Jan 17, 2011 at 06:24:15PM +0100, Tejun Heo wrote:
>>> On Mon, Jan 17, 2011 at 11:21:46AM -0600, James Bottomley wrote:
>>>> Expanders aren't SCSI devices either ... that means they don't
>>>> appear as visible to standard SCSI mechanisms like INQUIRY (SMP
>>>> isn't a SCSI protocol it's a SAS extension). They just appear as
>>>> part of the topology in the device tree.

>>> Ah, okay.  Then, things would be much simpler.  I was worrying about
>>> how it would map to SCSI layer including INQUIRY emulations and all
>>> those stuff.  Thanks for the info.

>> So, any thoughts on how and where to add the PMP to the devices
>> available for sending SCSI or ATA commands to?

>> I'm willing to try to add the required code, but I'm sure I'll
>> benefit from a few hints from the experts ...

> Well, I'm not sure I'd count as an expert on this piece:
> I haven't read the relevant ATA standards. 

> From what I understand, SES packets are encapsulated over 
> an ATA command for SEMB.

correct, just that it can be any protocol (it's just another
bus, usually i2c) with the common protocols being SES and
SAF_TE AFAIK

> In that case, it should be fairly simple to recognise
> this and present a SCSI device which simply encapsulates
> everything sent to it over this protocol. 

do you have any pointers for me how that is done for SCSI?

> That would allow the ses ULD to attach seamlessly.

any pointers to such an SES ULD (maybe for testing)?

thanks in advance,
Herbert

> James


> --
> To unsubscribe from this list: send the line "unsubscribe linux-ide" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: PMP and SEMB messages to SEP

[James Bottomley]

On Sun, 2011-04-03 at 19:39 +0200, Herbert Poetzl wrote:
> >> So, any thoughts on how and where to add the PMP to the devices
> >> available for sending SCSI or ATA commands to?
> 
> >> I'm willing to try to add the required code, but I'm sure I'll
> >> benefit from a few hints from the experts ...
> 
> > Well, I'm not sure I'd count as an expert on this piece:
> > I haven't read the relevant ATA standards. 
> 
> > From what I understand, SES packets are encapsulated over 
> > an ATA command for SEMB.
> 
> correct, just that it can be any protocol (it's just another
> bus, usually i2c) with the common protocols being SES and
> SAF_TE AFAIK
> 
> > In that case, it should be fairly simple to recognise
> > this and present a SCSI device which simply encapsulates
> > everything sent to it over this protocol. 
> 
> do you have any pointers for me how that is done for SCSI?

I don't think I understand the question.  It happens automatically for
anything part of the SCSI domain.  For SEMB you have to unwind the
encapsulation and expose the resulting device into the SCSI domain for
the SES ULD to attach, presumably as a libata target or LUN.  If you're
asking for basic documentation about how SCSI works, that's in
Documentation/scsi/ plus some of the kernel docbook stuff.

> > That would allow the ses ULD to attach seamlessly.
> 
> any pointers to such an SES ULD (maybe for testing)?

it's drivers/scsi/ses.c

James

Re: PMP and SEMB messages to SEP

[Jeff Garzik]

On 01/17/2011 11:18 AM, James Bottomley wrote:
> On Mon, 2011-01-17 at 16:40 +0100, Tejun Heo wrote:
>>> - the SiI3726 supports GPIO pins, which can be reached via
>>>    the General Status and Control Register [130] and accoring
>>>    to the docu, the Read/Write Port Multiplier command can
>>>    be used to read/write that register.
>>>
>>>    + how would I go about issuing such a command and where
>>>      should it be done? i.e. what about interference with
>>>      other commands? what about retrieving return values?
>>
>> The problem is that the PMP device itself is currently not allocated a
>> userland visible device, so it doesn't have any /dev/* node.  Hmmm...
>
> So perhaps it should be.  If you look at the equivalent topology on SAS,
> our expanders have a bsg device node precisely so that we can do this.
>
> That said, SAS expanders have a defined protocol (SAS Management
> Protocol) to talk to the outside world, so they are real visible objects
> always in our topology ... I'm not sure PMP has this ... it seems that
> all PMP visibility is an extension to the standard?

Yes, we do need a way to address these devices.

	Jeff