support for sata7 Streaming Feature Set?

support for sata7 Streaming Feature Set?

[Jan Wagner]

Hi,

anyone know if the now already somewhat old Streaming Feature Set of
ATA/ATAPI 7 is going to be implemented in the kernel ata functions?

According to one web site that contains hdreg.h
http://www.koders.com/c/fidCD7293464D782E48F93EEF8A71192F1BF28FC205.aspx
there's at least some kind of mention in that include file about streaming
feature set, kernel 2.6.10. However in 2.6.16 it seems to be gone again.
Any ideas if this will be implemented, or how to use it with e.g. hdparm
right now?

many thanks!
 - Jan

Re: support for sata7 Streaming Feature Set?

[Tejun Heo]

Jan Wagner wrote:
> Hi,
> 
> anyone know if the now already somewhat old Streaming Feature Set of
> ATA/ATAPI 7 is going to be implemented in the kernel ata functions?
> 
> According to one web site that contains hdreg.h
> http://www.koders.com/c/fidCD7293464D782E48F93EEF8A71192F1BF28FC205.aspx
> there's at least some kind of mention in that include file about streaming
> feature set, kernel 2.6.10. However in 2.6.16 it seems to be gone again.
> Any ideas if this will be implemented, or how to use it with e.g. hdparm
> right now?
> 

I don't think streaming feature set is something to be supported at 
kernel driver level.  The usage model doesn't fit with block interface. 
  If you want to use it, the best way would be issuing commands directly 
using sg.  What are you gonna use it for?

-- 
tejun

Re: support for sata7 Streaming Feature Set?

[Jan Wagner]

Hi and thanks for your response,

On Sun, 14 May 2006, Tejun Heo wrote:
> > anyone know if the now already somewhat old Streaming Feature Set of
> > ATA/ATAPI 7 is going to be implemented in the kernel ata functions?
> >
> > According to one web site that contains hdreg.h
> > http://www.koders.com/c/fidCD7293464D782E48F93EEF8A71192F1BF28FC205.aspx
> > there's at least some kind of mention in that include file about streaming
> > feature set, kernel 2.6.10. However in 2.6.16 it seems to be gone again.
> > Any ideas if this will be implemented, or how to use it with e.g. hdparm
> > right now?
>
> I don't think streaming feature set is something to be supported at
> kernel driver level.  The usage model doesn't fit with block interface.

I'm definitely not a kernel guru or into its internals, but IMHO ATA/ATAPI
specifications should all also be supported in the kernel or kernel
module, for compliance, or?

The block device's ioctl could have a "data reliability setting"
extension, specifying either the error recovery time limits or for
enabling continuous read/write control (used to return/use partially
correct data) which are part of the ATA Streaming Feature Set.

I.e. an adjustable minimum acceptable data reliability level for block
devices, which can e.g. be relaxed down from a default 100%.

>   If you want to use it, the best way would be issuing commands directly
> using sg.

Maybe yes, that, or hdparm, but it seems like a horrible hack :) And sg
being for generic SCSI, I'm not sure how well ATA-7 fits in. At least,
the current debian sg-tools, and commands like 'sg_opcodes /dev/sda'
return "Fixed format, current;  Sense key: Illegal Request", "Additional
sense: Invalid command operation code" for those SATA disks I tried.
Doesn't look good for sg useability, AFAICT.

> What are you gonna use it for?

To record or play back real-time continuous streamed data that is not
error-critical but delay critical, from/to a bidirectional data
aquisition card at ~1Gbit/s over longer time spans.

Direct kernel device support for the feature set could also be very useful
for linux projects like the Digital Video Recorder and Video Disk
Recorder. And seek/stutter free video playback from DVD/ATAPI (scratched
disks, for example) or video editing. Etc.

 - Jan

Re: support for sata7 Streaming Feature Set?

[Lennart Sorensen]

On Wed, May 17, 2006 at 11:30:55AM +0300, Jan Wagner wrote:
> Maybe yes, that, or hdparm, but it seems like a horrible hack :) And sg
> being for generic SCSI, I'm not sure how well ATA-7 fits in. At least,
> the current debian sg-tools, and commands like 'sg_opcodes /dev/sda'
> return "Fixed format, current;  Sense key: Illegal Request", "Additional
> sense: Invalid command operation code" for those SATA disks I tried.
> Doesn't look good for sg useability, AFAICT.

Just because sgtools doesn't work with it doesn't mean there aren't
commands you can send through sg to work with it.

> To record or play back real-time continuous streamed data that is not
> error-critical but delay critical, from/to a bidirectional data
> aquisition card at ~1Gbit/s over longer time spans.

Do you know of a disk that can handle 1Gbit/s to the platter?  Or are
you planning to stripe this across multiple disks?

I would think a controller on a fast enough bus (plain PCI isn't going
to handle it), with enough drives in a raid setup of the right type
should probably handle it.  Might need to do a filesystem specially
designed for the streaming needs rather than general purpose file
storage.

> Direct kernel device support for the feature set could also be very useful
> for linux projects like the Digital Video Recorder and Video Disk
> Recorder. And seek/stutter free video playback from DVD/ATAPI (scratched
> disks, for example) or video editing. Etc.

Can you tell a DVD drive to stop retrying?  Perhaps you can.  I know
some of the retries are in software.

Len Sorensen

Re: support for sata7 Streaming Feature Set?

[Jan Wagner]

Hi,

On Wed, 17 May 2006, Lennart Sorensen wrote:
> > To record or play back real-time continuous streamed data that is not
> > error-critical but delay critical, from/to a bidirectional data
> > aquisition card at ~1Gbit/s over longer time spans.
>
> Do you know of a disk that can handle 1Gbit/s to the platter?  Or are
> you planning to stripe this across multiple disks?
> I would think a controller on a fast enough bus (plain PCI isn't going
> to handle it), with enough drives in a raid setup of the right type
> should probably handle it.  Might need to do a filesystem specially
> designed for the streaming needs rather than general purpose file
> storage.

Yes, multiple disks striped in RAID-0, 4 x DiamondMax 10 300GB, ext2 fs.
On a cheapish Dell OptiPlex GX620 that uses Intel 945G. 1.6Gbit/s write
to disk while reading PCI(-X) goes just fine, though of course gets
slower near disk ends.

Streaming filesystems, yes, that's what I'd like to evaluate next vs
ext2 once the Samsung SP2504C disks with Streaming Feature Set support
arrive.

> > Direct kernel device support for the feature set could also be very useful
> > for linux projects like the Digital Video Recorder and Video Disk
> > Recorder. And seek/stutter free video playback from DVD/ATAPI (scratched
> > disks, for example) or video editing. Etc.
>
> Can you tell a DVD drive to stop retrying?  Perhaps you can.  I know
> some of the retries are in software.

Ok, probably yes, this is also something to be done outside of kernel.

However this guy over here
 http://www.ussg.iu.edu/hypermail/linux/kernel/0411.3/0338.html
has written speedcontrol.c for SET STREAMING (linked pdf there, page
457), which is somewhat similar to the commands in the Streaming feature
set. One quote: "In my opinion it would make sence to also enhance the
kernel function cdrom_select_speed (linux/drivers/ide/ide-cd.c), so that
this function works also for "newer" DVD-drives." etc

But OTOH also Tejun's points about that this Streaming set should be
implemented in userland are quite valid. And I don't know where exactly
such functionality could be added to the kernel i.e. where I would propose
such a feature to be added; it was a thought there might be some suitable
place and people here would know where to add ;-))

But actually after googling more, XFS file system seems to have something
like Streaming feature set support, or at least provisions for adding such
commands, as it has a guaranteed-rate I/O feature
  http://en.wikipedia.org/wiki/XFS#Guaranteed_rate_I.2FO

So I guess XFS code is the better place to start thinking about Streaming
feature set implementation.

Many thanks for your input! :-))

 - Jan

Re: support for sata7 Streaming Feature Set?

[Tejun Heo]

Jan Wagner wrote:
> Hi and thanks for your response,
> 
> On Sun, 14 May 2006, Tejun Heo wrote:
>>> anyone know if the now already somewhat old Streaming Feature Set of
>>> ATA/ATAPI 7 is going to be implemented in the kernel ata functions?
>>>
>>> According to one web site that contains hdreg.h
>>> http://www.koders.com/c/fidCD7293464D782E48F93EEF8A71192F1BF28FC205.aspx
>>> there's at least some kind of mention in that include file about streaming
>>> feature set, kernel 2.6.10. However in 2.6.16 it seems to be gone again.
>>> Any ideas if this will be implemented, or how to use it with e.g. hdparm
>>> right now?
>> I don't think streaming feature set is something to be supported at
>> kernel driver level.  The usage model doesn't fit with block interface.
> 
> I'm definitely not a kernel guru or into its internals, but IMHO ATA/ATAPI
> specifications should all also be supported in the kernel or kernel
> module, for compliance, or?

No, not really.  If things can be done outside of kernel, we like to 
keep them there.  e.g.  SMART is implemented in the userland and it fits 
there.  Kernel only has to provide mechanisms to enable such implementation.

> The block device's ioctl could have a "data reliability setting"
> extension, specifying either the error recovery time limits or for
> enabling continuous read/write control (used to return/use partially
> correct data) which are part of the ATA Streaming Feature Set.
> 
> I.e. an adjustable minimum acceptable data reliability level for block
> devices, which can e.g. be relaxed down from a default 100%.

Streaming / relaxed reliablity is a very specialized feature requiring 
very specialized software to drive it.  I can't think of much use in the 
generic vm/fs/block framework.  So, I don't see it happening in the kernel.

>>   If you want to use it, the best way would be issuing commands directly
>> using sg.
> 
> Maybe yes, that, or hdparm, but it seems like a horrible hack :) And sg
> being for generic SCSI, I'm not sure how well ATA-7 fits in. At least,
> the current debian sg-tools, and commands like 'sg_opcodes /dev/sda'
> return "Fixed format, current;  Sense key: Illegal Request", "Additional
> sense: Invalid command operation code" for those SATA disks I tried.
> Doesn't look good for sg useability, AFAICT.

No, it's not a horrible hack.  Again, no one thinks smartd is a horrible 
hack.  Even core things like irqbalancing and CPU speeding down are 
controlled from userland.  Being in the userland is a good thing - it's 
much safer & more convenient/flexible out there.

>> What are you gonna use it for?
> 
> To record or play back real-time continuous streamed data that is not
> error-critical but delay critical, from/to a bidirectional data
> aquisition card at ~1Gbit/s over longer time spans.

One thing to think about before supporting streaming from/to harddisks 
from userland is how to make data flow efficiently from userland to 
kernel and back.  But, no matter what, kernel <-> userland usually 
involves one data copy, so I don't think making sg similarly efficient 
would be too difficult (it might be already).

> Direct kernel device support for the feature set could also be very useful
> for linux projects like the Digital Video Recorder and Video Disk
> Recorder. And seek/stutter free video playback from DVD/ATAPI (scratched
> disks, for example) or video editing. Etc.

Why direct kernel device support necessary for such things?  What kind 
of interface are you proposing?  I can't think of anything better than 
libatastreaming (or whatever, just focus on the leading 'lib'), which 
uses sg to issue commands and manage everything.

-- 
tejun

Re: support for sata7 Streaming Feature Set?

[Mark Lord]

Tejun Heo wrote:
> Jan Wagner wrote:
>> Hi and thanks for your response,
>>
>> On Sun, 14 May 2006, Tejun Heo wrote:
>>>> anyone know if the now already somewhat old Streaming Feature Set of
>>>> ATA/ATAPI 7 is going to be implemented in the kernel ata functions?
>>>>
>>>> According to one web site that contains hdreg.h
>>>> http://www.koders.com/c/fidCD7293464D782E48F93EEF8A71192F1BF28FC205.aspx 
...
That link just gives me a search page.  Got a better link?


> No, it's not a horrible hack.  Again, no one thinks smartd is a horrible 
> hack.  Even core things like irqbalancing and CPU speeding down are 
> controlled from userland.  Being in the userland is a good thing - it's 
> much safer & more convenient/flexible out there.
> 
>>> What are you gonna use it for?
>>
>> To record or play back real-time continuous streamed data that is not
>> error-critical but delay critical, from/to a bidirectional data
>> aquisition card at ~1Gbit/s over longer time spans.
..
Other uses include tons of embedded systems using Linux,
primarily those that handle any kind of A/V record/playback
(think "Tivo", MP3 player, video players, ...).
These are about as rare as a fileserver, when counting the installed base.

I think that the ATA streaming feature set cannot be done
efficiently (which is the whole point of it!) without some kernel support.

The device driver has to know about it, at a minimum so that it can
select a different EH protocol for the streams.  Which in turn means
that the streaming commands should be known to the driver as well.

But how to handle it all nicely is the real question.
A new block driver, if libata cannot handle it?

Cheers

Re: support for sata7 Streaming Feature Set?

[Jeff Garzik]

Mark Lord wrote:
> The device driver has to know about it, at a minimum so that it can
> select a different EH protocol for the streams.  Which in turn means
> that the streaming commands should be known to the driver as well.

Different taskfile protocol, you mean?


> But how to handle it all nicely is the real question.
> A new block driver, if libata cannot handle it?

I seriously doubt writing a whole new ATA driver subsystem will fly :)

	Jeff

Re: support for sata7 Streaming Feature Set?

[Tejun Heo]

Jeff Garzik wrote:
> Mark Lord wrote:
>> The device driver has to know about it, at a minimum so that it can
>> select a different EH protocol for the streams.  Which in turn means
>> that the streaming commands should be known to the driver as well.
> 
> Different taskfile protocol, you mean?

I haven't checked all the docs and codes thoroughly but I don't see a 
need for new protocol or anything.  When a streaming command fails due 
to failing to meet timing constraints, it fails w/ device error.  sg can 
adjust retry count and the device error will be reported without much 
recovery action (only revalidation).  If the device fails due to some 
other reasons (say HSM violation), it needs full EH no matter what. 
Without full EH, it becomes completely unusable.

>> But how to handle it all nicely is the real question.
>> A new block driver, if libata cannot handle it?
> 
> I seriously doubt writing a whole new ATA driver subsystem will fly :)

All I can see are little extensions to sg interface and maybe libata.  I 
don't think it needs full-blown in-kernel driver.  However, to use this 
feature with a filesystem, we would need to build a block map of the 
file to use.  I think such feature is already provided and used by 
LILO/GRUB kinds of things.

-- 
tejun

Re: support for sata7 Streaming Feature Set?

[Mark Lord]

Jeff Garzik wrote:
> 
> I seriously doubt writing a whole new ATA driver subsystem will fly :)

Agreed.  ;)

Re: support for sata7 Streaming Feature Set?

[Jeff Garzik]

Tejun Heo wrote:
> One thing to think about before supporting streaming from/to harddisks 
> from userland is how to make data flow efficiently from userland to 
> kernel and back.  But, no matter what, kernel <-> userland usually 
> involves one data copy, so I don't think making sg similarly efficient 
> would be too difficult (it might be already).

Actually, the kernel usually maps userland pages, eliminating the need 
for a copy.  write(2) may have copied data into that page originally, 
but mmap(2) need not have.

	Jeff

Re: support for sata7 Streaming Feature Set?

[Tejun Heo]

Jeff Garzik wrote:
> Tejun Heo wrote:
>> One thing to think about before supporting streaming from/to harddisks 
>> from userland is how to make data flow efficiently from userland to 
>> kernel and back.  But, no matter what, kernel <-> userland usually 
>> involves one data copy, so I don't think making sg similarly efficient 
>> would be too difficult (it might be already).
> 
> Actually, the kernel usually maps userland pages, eliminating the need 
> for a copy.  write(2) may have copied data into that page originally, 
> but mmap(2) need not have.

Yeap, to achieve high streaming rate, it would be best to have 
preallocated ring buffer and ring pointers.  If this high-bw streaming 
thing becomes common, we can add it to sg, I guess.

-- 
tejun