SATA hotplug from the user side ?

SATA hotplug from the user side ?

[Soeren Sonnenburg]

Dear all,

I'd like to try out SATA hotplugging using a SIL3114. Though I was
harvesting the web, I could not find any useful information how this is
done in practice.

Well I realized that I can still use scsiadd to print and remove
devices, e.g.:

# scsiadd -p

Attached devices:
Host: scsi2 Channel: 00 Id: 00 Lun: 00
  Vendor: ATA      Model: ST3400832AS      Rev: 3.01
  Type:   Direct-Access                    ANSI SCSI revision: 05
Host: scsi3 Channel: 00 Id: 00 Lun: 00
  Vendor: ATA      Model: ST3400620AS      Rev: 3.AA
  Type:   Direct-Access                    ANSI SCSI revision: 05

# scsiadd -r 3 0 0 0

Is this all one has to do for hotplugging ? I am asking as I find this
in dmesg when I do so (2.6.19.* kernel):

Synchronizing SCSI cache for disk sdb: 
ata4.00: disabled
ata4: exception Emask 0x10 SAct 0x0 SErr 0x10000 action 0x2 frozen
ata4: hard resetting port
ata4: SATA link down (SStatus 0 SControl 310)
ata4: EH complete
ata4: exception Emask 0x10 SAct 0x0 SErr 0x50000 action 0x2 frozen
ata4: hard resetting port
ata4: port is slow to respond, please be patient (Status 0xff)
ata4: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
ata4.00: ATA-7, max UDMA/133, 781422768 sectors: LBA48 NCQ (depth 0/32)
ata4.00: configured for UDMA/100
ata4: EH complete
scsi 3:0:0:0: rejecting I/O to dead device
scsi 3:0:0:0: rejecting I/O to dead device


Soeren
-- 
For the one fact about the future of which we can be certain is that it
will be utterly fantastic. -- Arthur C. Clarke, 1962

Re: SATA hotplug from the user side ?

[Jeff Garzik]

Soeren Sonnenburg wrote:
> Dear all,
> 
> I'd like to try out SATA hotplugging using a SIL3114. Though I was
> harvesting the web, I could not find any useful information how this is
> done in practice.
> 
> Well I realized that I can still use scsiadd to print and remove
> devices, e.g.:

For SIL3114, you shouldn't have to run any commands at all.  It should 
notice when you yank the cable, or plug in a new device.

	Jeff

Re: SATA hotplug from the user side ?

[Soeren Sonnenburg]

On Fri, 2007-01-12 at 12:04 -0500, Jeff Garzik wrote:
> Soeren Sonnenburg wrote:
> > Dear all,
> > 
> > I'd like to try out SATA hotplugging using a SIL3114. Though I was
> > harvesting the web, I could not find any useful information how this is
> > done in practice.
> > 
> > Well I realized that I can still use scsiadd to print and remove
> > devices, e.g.:
> 
> For SIL3114, you shouldn't have to run any commands at all.  It should 
> notice when you yank the cable, or plug in a new device.


It is true it detects a removal and newly plugged devices immediately...
However it still prints warnings and errors that it could not
synchronize SCSI cache for the disks. Then it prints regular 'rejects
I/O to dead device' warning messages and on replugging the disks puts
them to the next free sd device (e.g. sdc -> sdd).

These messages sound eval - so now the question is should I care ?
( On the other hand it did not crash the machine )

What follows is a change between to sata drives attached to port 4/5 of
the sil (ata5/ata6 here):

ata6: exception Emask 0x10 SAct 0x0 SErr 0x10000 action 0x2 frozen
ata6: hard resetting port
ata6: SATA link down (SStatus 0 SControl 310)
ata6: failed to recover some devices, retrying in 5 secs
ata5: exception Emask 0x10 SAct 0x0 SErr 0x10000 action 0x2 frozen
ata5: hard resetting port
ata5: SATA link down (SStatus 0 SControl 310)
ata5: failed to recover some devices, retrying in 5 secs
ata6: hard resetting port
ata6: SATA link down (SStatus 0 SControl 310)
ata6: failed to recover some devices, retrying in 5 secs
ata5: hard resetting port
ata5: SATA link down (SStatus 0 SControl 310)
ata5: failed to recover some devices, retrying in 5 secs
ata6: hard resetting port
ata6: SATA link down (SStatus 0 SControl 310)
ata6.00: disabled
ata6: EH complete
ata6.00: detaching (SCSI 5:0:0:0)
Synchronizing SCSI cache for disk sdd: 
FAILED
  status = 0, message = 00, host = 4, driver = 00
  <6>ata5: hard resetting port
ata5: SATA link down (SStatus 0 SControl 310)
ata5.00: disabled
ata5: EH complete
ata5.00: detaching (SCSI 4:0:0:0)
Synchronizing SCSI cache for disk sdc: 
FAILED
  status = 0, message = 00, host = 4, driver = 00
  <3>ata6: exception Emask 0x10 SAct 0x0 SErr 0x50000 action 0x2 frozen
ata6: hard resetting port
ata6: port is slow to respond, please be patient (Status 0xff)
ata6: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
ata6.00: ATA-7, max UDMA/133, 1465149168 sectors: LBA48 NCQ (depth 0/32)
ata6.00: configured for UDMA/100
ata6: EH complete
scsi 5:0:0:0: Direct-Access     ATA      ST3750640AS      3.AA PQ: 0
ANSI: 5
SCSI device sdf: 1465149168 512-byte hdwr sectors (750156 MB)
sdf: Write Protect is off
sdf: Mode Sense: 00 3a 00 00
SCSI device sdf: drive cache: write back
SCSI device sdf: 1465149168 512-byte hdwr sectors (750156 MB)
sdf: Write Protect is off
sdf: Mode Sense: 00 3a 00 00
SCSI device sdf: drive cache: write back
 sdf: unknown partition table
sd 5:0:0:0: Attached scsi disk sdf
sd 5:0:0:0: Attached scsi generic sg2 type 0
scsi 4:0:0:0: rejecting I/O to dead device
scsi 4:0:0:0: rejecting I/O to dead device
scsi 5:0:0:0: rejecting I/O to dead device
scsi 5:0:0:0: rejecting I/O to dead device
ata5: exception Emask 0x10 SAct 0x0 SErr 0x50000 action 0x2 frozen
ata5: hard resetting port
ata5: port is slow to respond, please be patient (Status 0xff)
ata5: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
ata5.00: ATA-7, max UDMA/133, 1465149168 sectors: LBA48 NCQ (depth 0/32)
ata5.00: configured for UDMA/100
ata5: EH complete
scsi 4:0:0:0: Direct-Access     ATA      ST3750640AS      3.AA PQ: 0
ANSI: 5
SCSI device sdg: 1465149168 512-byte hdwr sectors (750156 MB)
sdg: Write Protect is off
sdg: Mode Sense: 00 3a 00 00
SCSI device sdg: drive cache: write back
SCSI device sdg: 1465149168 512-byte hdwr sectors (750156 MB)
sdg: Write Protect is off
sdg: Mode Sense: 00 3a 00 00
SCSI device sdg: drive cache: write back
 sdg: unknown partition table
sd 4:0:0:0: Attached scsi disk sdg
sd 4:0:0:0: Attached scsi generic sg3 type 0

Best,
Soeren
-- 
For the one fact about the future of which we can be certain is that it
will be utterly fantastic. -- Arthur C. Clarke, 1962

Re: SATA hotplug from the user side ?

[Tejun Heo]

Soeren Sonnenburg wrote:
> It is true it detects a removal and newly plugged devices immediately...
> However it still prints warnings and errors that it could not
> synchronize SCSI cache for the disks. Then it prints regular 'rejects
> I/O to dead device' warning messages and on replugging the disks puts
> them to the next free sd device (e.g. sdc -> sdd).

You need to stop using the devices before unplugging.  If you have no
pending IO to the device, there won't be 'rejects IO to dead device'
messages.  You can ignore the SCSI cache sync failure if the device is
properly closed before being unplugged.

> These messages sound eval - so now the question is should I care ?
> ( On the other hand it did not crash the machine )

So, no, you don't really have to care.  Just make sure the device is
unmounted prior to unplugging.

-- 
tejun

Re: SATA hotplug from the user side ?

[Soeren Sonnenburg]

On Sat, 2007-01-13 at 10:55 +0900, Tejun Heo wrote:
> Soeren Sonnenburg wrote:
> > It is true it detects a removal and newly plugged devices immediately...
> > However it still prints warnings and errors that it could not
> > synchronize SCSI cache for the disks. Then it prints regular 'rejects
> > I/O to dead device' warning messages and on replugging the disks puts
> > them to the next free sd device (e.g. sdc -> sdd).
> 
> You need to stop using the devices before unplugging.  If you have no
> pending IO to the device, there won't be 'rejects IO to dead device'
> messages.  You can ignore the SCSI cache sync failure if the device is
> properly closed before being unplugged.

Jeff & Tejun thanks *a lot* for clarifying this. I am quite happy to see
that this is working very reliably!

> > These messages sound eval - so now the question is should I care ?
> > ( On the other hand it did not crash the machine )
> 
> So, no, you don't really have to care.  Just make sure the device is
> unmounted prior to unplugging.

OK, but then this really should be in the SATA hotplug FAQ (or can one
fix this somehow?)... No user will ignore messages like this. What is
especially annoying is that udev on the first remove/insert cycle
created a new device node so the disk became /dev/sde (was /dev/sdd):
dmesg output of reinserting the disk 2 times follows:

<remove /dev/sdd>
ata4: exception Emask 0x10 SAct 0x0 SErr 0x10000 action 0x2 frozen
ata4: hard resetting port
ata4: SATA link down (SStatus 0 SControl 310)
ata4: failed to recover some devices, retrying in 5 secs
ata4: hard resetting port
ata4: SATA link down (SStatus 0 SControl 310)
ata4: failed to recover some devices, retrying in 5 secs
ata4: hard resetting port
ata4: SATA link down (SStatus 0 SControl 310)
ata4.00: disabled
ata4: EH complete
ata4.00: detaching (SCSI 3:0:0:0)
Synchronizing SCSI cache for disk sdd: 
FAILED
  status = 0, message = 00, host = 4, driver = 00
  <3>ata4: exception Emask 0x10 SAct 0x0 SErr 0x50000 action 0x2 frozen
ata4: hard resetting port
ata4: COMRESET failed (device not ready)
ata4: hardreset failed, retrying in 5 secs
ata4: hard resetting port
ata4: COMRESET failed (device not ready)
ata4: hardreset failed, retrying in 5 secs
ata4: hard resetting port

<insert>
ata4: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
ata4.00: ATA-7, max UDMA/133, 1465149168 sectors: LBA48 NCQ (depth 0/32)
ata4.00: configured for UDMA/100
ata4: EH complete
scsi 3:0:0:0: Direct-Access     ATA      ST3750640AS      3.AA PQ: 0
ANSI: 5
SCSI device sde: 1465149168 512-byte hdwr sectors (750156 MB)
sde: Write Protect is off
sde: Mode Sense: 00 3a 00 00
SCSI device sde: drive cache: write back
SCSI device sde: 1465149168 512-byte hdwr sectors (750156 MB)
sde: Write Protect is off
sde: Mode Sense: 00 3a 00 00
SCSI device sde: drive cache: write back
 sde: unknown partition table
sd 3:0:0:0: Attached scsi disk sde
sd 3:0:0:0: Attached scsi generic sg3 type 0

<remove now /dev/sde>
ata4: exception Emask 0x10 SAct 0x0 SErr 0x10000 action 0x2 frozen
ata4: hard resetting port
ata4: SATA link down (SStatus 0 SControl 310)
ata4: failed to recover some devices, retrying in 5 secs
ata4: hard resetting port
ata4: SATA link down (SStatus 0 SControl 310)
ata4: failed to recover some devices, retrying in 5 secs
ata4: hard resetting port
ata4: SATA link down (SStatus 0 SControl 310)
ata4.00: disabled
ata4: EH complete
ata4.00: detaching (SCSI 3:0:0:0)
Synchronizing SCSI cache for disk sde: 
FAILED
  status = 0, message = 00, host = 4, driver = 00
  <3>ata4: exception Emask 0x10 SAct 0x0 SErr 0x50000 action 0x2 frozen
ata4: hard resetting port
ata4: COMRESET failed (device not ready)
ata4: hardreset failed, retrying in 5 secs
ata4: hard resetting port
ata4: COMRESET failed (device not ready)
ata4: hardreset failed, retrying in 5 secs
ata4: hard resetting port

<insert>
ata4: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
ata4.00: ATA-7, max UDMA/133, 1465149168 sectors: LBA48 NCQ (depth 0/32)
ata4.00: configured for UDMA/100
ata4: EH complete
scsi 3:0:0:0: Direct-Access     ATA      ST3750640AS      3.AA PQ: 0
ANSI: 5
SCSI device sde: 1465149168 512-byte hdwr sectors (750156 MB)
sde: Write Protect is off
sde: Mode Sense: 00 3a 00 00
SCSI device sde: drive cache: write back
SCSI device sde: 1465149168 512-byte hdwr sectors (750156 MB)
sde: Write Protect is off
sde: Mode Sense: 00 3a 00 00
SCSI device sde: drive cache: write back
 sde: unknown partition table
sd 3:0:0:0: Attached scsi disk sde
sd 3:0:0:0: Attached scsi generic sg3 type 0

remains /dev/sde ... 

Soeren
-- 
Sometimes, there's a moment as you're waking, when you become aware of
the real world around you, but you're still dreaming.

Re: SATA hotplug from the user side ?

[Tejun Heo]

Soeren Sonnenburg wrote:
> Jeff & Tejun thanks *a lot* for clarifying this. I am quite happy to see
> that this is working very reliably!

You're welcome.  :-)

>>> These messages sound eval - so now the question is should I care ?
>>> ( On the other hand it did not crash the machine )
>> So, no, you don't really have to care.  Just make sure the device is
>> unmounted prior to unplugging.
> 
> OK, but then this really should be in the SATA hotplug FAQ (or can one
> fix this somehow?)... No user will ignore messages like this. What is

Yeah, probably.  Care to submit patch to FAQ to Jeff?

> especially annoying is that udev on the first remove/insert cycle
> created a new device node so the disk became /dev/sde (was /dev/sdd):
> dmesg output of reinserting the disk 2 times follows:

That's because something is still holding onto /dev/sdd.  The device
remains dangled until all references to it are gone.  /dev/sdd is still
lingering when the drive is hotplugged; thus, it gets assigned the next
device.

I think this is best solved by udev.  Think of /dev/sdX as kernel
internal name which may change from time to time.  Mount devices with
LABEL's and use udev-provided persistent names to access removable
devices.  Also, libata can be improved to tell udev that certain ports
are external, I think.

Thanks.

-- 
tejun

Re: SATA hotplug from the user side ?

[Soeren Sonnenburg]

On Mon, 2007-01-15 at 11:21 +0900, Tejun Heo wrote:
> Soeren Sonnenburg wrote:
> > Jeff & Tejun thanks *a lot* for clarifying this. I am quite happy to see
> > that this is working very reliably!
> 
> You're welcome.  :-)
> 
> >>> These messages sound eval - so now the question is should I care ?
> >>> ( On the other hand it did not crash the machine )
> >> So, no, you don't really have to care.  Just make sure the device is
> >> unmounted prior to unplugging.
> > 
> > OK, but then this really should be in the SATA hotplug FAQ (or can one
> > fix this somehow?)... No user will ignore messages like this. What is
> 
> Yeah, probably.  Care to submit patch to FAQ to Jeff?

OK, how about this (please especially check the non SIL part):

SATA Hotplug from the User Side

- For SIL3114 and SIL3124 you don't have to run any commands at all. It
should notice when you yank the cable, or plug in a new device. All you
have to do is to stop using the devices before unplugging, e.g. unmount
partitions on the disk or remove the disk from a dm setup). One can
validate which disks are attached using ``scsiadd -p''. Note that the
device name may change from e.g. /dev/sdd to /dev/sde on a
remove/reinsert cycle (this can be fixed by using udev-provided
persistent names). Also note that it is perfectly normal to see messages
like this in dmesg:

<removing /dev/sdd>

ata4: exception Emask 0x10 SAct 0x0 SErr 0x10000 action 0x2 frozen
ata4: hard resetting port
ata4: SATA link down (SStatus 0 SControl 310)
ata4: failed to recover some devices, retrying in 5 secs
ata4: hard resetting port
ata4: SATA link down (SStatus 0 SControl 310)
ata4: failed to recover some devices, retrying in 5 secs
ata4: hard resetting port
ata4: SATA link down (SStatus 0 SControl 310)
ata4.00: disabled
ata4: EH complete
ata4.00: detaching (SCSI 3:0:0:0)
Synchronizing SCSI cache for disk sdd: 
FAILED
  status = 0, message = 00, host = 4, driver = 00
  <3>ata4: exception Emask 0x10 SAct 0x0 SErr 0x50000 action 0x2 frozen
ata4: hard resetting port
ata4: COMRESET failed (device not ready)
ata4: hardreset failed, retrying in 5 secs
ata4: hard resetting port
ata4: COMRESET failed (device not ready)
ata4: hardreset failed, retrying in 5 secs
ata4: hard resetting port

<reinserting the disk>
ata4: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
ata4.00: ATA-7, max UDMA/133, 1465149168 sectors: LBA48 NCQ (depth 0/32)
ata4.00: configured for UDMA/100
ata4: EH complete
scsi 3:0:0:0: Direct-Access     ATA      ST3750640AS      3.AA PQ: 0
ANSI: 5
SCSI device sde: 1465149168 512-byte hdwr sectors (750156 MB)
sde: Write Protect is off
sde: Mode Sense: 00 3a 00 00
SCSI device sde: drive cache: write back
SCSI device sde: 1465149168 512-byte hdwr sectors (750156 MB)
sde: Write Protect is off
sde: Mode Sense: 00 3a 00 00
SCSI device sde: drive cache: write back
 sde: unknown partition table
sd 3:0:0:0: Attached scsi disk sde
sd 3:0:0:0: Attached scsi generic sg3 type 0

However if you happen to see messages like

scsi 3:0:0:0: rejecting I/O to dead device
scsi 4:0:0:0: rejecting I/O to dead device
scsi 5:0:0:0: rejecting I/O to dead device

you did not stop using the devices before unplugging (check that you
unmounted all partitions, removed the disk from a raid array, dmsetup,
cryptsetup).  If you have no pending IO to the device, there won't be
'rejects IO to dead device' messages.

- For other chipsets one in addition to stop using the device before
unplugging has to call scsiadd -r to fully remove it, e.g. in the
following example the disk on scsi host 3 channel 0 id 0 lun 0 will be
removed, then on reinserting a disk call scsiadd -s :

# scsiadd -p

Attached devices:
Host: scsi2 Channel: 00 Id: 00 Lun: 00
  Vendor: ATA      Model: ST3400832AS      Rev: 3.01
  Type:   Direct-Access                    ANSI SCSI revision: 05
Host: scsi3 Channel: 00 Id: 00 Lun: 00
  Vendor: ATA      Model: ST3400620AS      Rev: 3.AA
  Type:   Direct-Access                    ANSI SCSI revision: 05

# scsiadd -r 3 0 0 0
# scsiadd -s

Soeren
-- 
Sometimes, there's a moment as you're waking, when you become aware of
the real world around you, but you're still dreaming.

Re: SATA hotplug from the user side ?

[Henrique de Moraes Holschuh]

On Mon, 22 Jan 2007, Soeren Sonnenburg wrote:
> - For SIL3114 and SIL3124 you don't have to run any commands at all. It
> should notice when you yank the cable, or plug in a new device. All you
> have to do is to stop using the devices before unplugging, e.g. unmount
> partitions on the disk or remove the disk from a dm setup). One can
> validate which disks are attached using ``scsiadd -p''. Note that the
> device name may change from e.g. /dev/sdd to /dev/sde on a
> remove/reinsert cycle (this can be fixed by using udev-provided
> persistent names). Also note that it is perfectly normal to see messages
> like this in dmesg:

Does SATA electrical conector keying let the disk firmware unload heads
before the user manages to pull it out enough to sever power?  If it does
not, the drive will do an emergency head unload, which is not good and will
likely reduce the drive's lifetime.

Using hdparm -Y before the unplug, or scsiadd -r (on a kernel that has
Tejun's new patch to optionally issue an START_STOP_UNIT to the SCSI device
enabled) is probably a good idea.  Unless it is a shared SATA port (I don't
know if such a thing exists yet) and another box is talking to the disk,
etc.

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh

Re: SATA hotplug from the user side ?

[Tejun Heo]

Henrique de Moraes Holschuh wrote:
> Does SATA electrical conector keying let the disk firmware unload
> heads before the user manages to pull it out enough to sever power?

I don't think so.

> If it does not, the drive will do an emergency head unload, which is
> not good and will likely reduce the drive's lifetime.

Probably.

> Using hdparm -Y before the unplug, or scsiadd -r (on a kernel that
> has Tejun's new patch to optionally issue an START_STOP_UNIT to the
> SCSI device enabled) is probably a good idea.  Unless it is a shared
> SATA port (I don't know if such a thing exists yet) and another box
> is talking to the disk, etc.

Agreed.  But it would be *much* better if all these can be taken care of
by hald and its minions.  Such that the user can just tell the system
that the hdd is going to be removed and all these dirty tricks are done
automagically.

-- 
tejun

Re: SATA hotplug from the user side ?

[Henrique de Moraes Holschuh]

On Tue, 23 Jan 2007, Tejun Heo wrote:
> Henrique de Moraes Holschuh wrote:
> > Does SATA electrical conector keying let the disk firmware unload
> > heads before the user manages to pull it out enough to sever power?
> 
> I don't think so.

Heh, thought as much. (Good) SCSI hotswap bays notice you pulled the disk
release lever and issue an START_STOP_UNIT by themselves to the disk well
before you have time to start pulling the disk out.  I wonder if the SATA
ones do (I kind of doubt that, SATA seems to attract the el-cheapo, el-crapo
crowd of manufacturers).

> > If it does not, the drive will do an emergency head unload, which is
> > not good and will likely reduce the drive's lifetime.
> 
> Probably.

So, that means it should be explained in the docs that you are to stop the
disk first, if you can.

> > Using hdparm -Y before the unplug, or scsiadd -r (on a kernel that
> > has Tejun's new patch to optionally issue an START_STOP_UNIT to the
> > SCSI device enabled) is probably a good idea.  Unless it is a shared
> > SATA port (I don't know if such a thing exists yet) and another box
> > is talking to the disk, etc.
> 
> Agreed.  But it would be *much* better if all these can be taken care of
> by hald and its minions.  Such that the user can just tell the system
> that the hdd is going to be removed and all these dirty tricks are done
> automagically.

Even if hald does this automatically, it would still be a very good idea to
document the proper sequence, IMO...

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh

Re: SATA hotplug from the user side ?

[Tejun Heo]

Henrique de Moraes Holschuh wrote:
> On Tue, 23 Jan 2007, Tejun Heo wrote:
>> Henrique de Moraes Holschuh wrote:
>>> Does SATA electrical conector keying let the disk firmware unload
>>> heads before the user manages to pull it out enough to sever power?
>> I don't think so.
> 
> Heh, thought as much. (Good) SCSI hotswap bays notice you pulled the disk
> release lever and issue an START_STOP_UNIT by themselves to the disk well
> before you have time to start pulling the disk out.  I wonder if the SATA
> ones do (I kind of doubt that, SATA seems to attract the el-cheapo, el-crapo
> crowd of manufacturers).

ahci controller spec has places for such thing but I've never seen it
actually implemented.  I'm thinking about a CLI tool to list & control
libata devices including hotplugging, NCQ and other stuff.  Eventually,
it would be really nice to give the user/admin easy gui/web/whatever
tool to watch and control ATA devices.

>>> If it does not, the drive will do an emergency head unload, which is
>>> not good and will likely reduce the drive's lifetime.
>> Probably.
> 
> So, that means it should be explained in the docs that you are to stop the
> disk first, if you can.

Yeap, agreed.

> Even if hald does this automatically, it would still be a very good idea to
> document the proper sequence, IMO...

Agreed.

-- 
tejun

Re: SATA hotplug from the user side ?

[Soeren Sonnenburg]

On Wed, 2007-01-24 at 11:07 +0900, Tejun Heo wrote:
> Henrique de Moraes Holschuh wrote:
> > On Tue, 23 Jan 2007, Tejun Heo wrote:
> >> Henrique de Moraes Holschuh wrote:
> >>> Does SATA electrical conector keying let the disk firmware unload
> >>> heads before the user manages to pull it out enough to sever power?
> >> I don't think so.
[...]
> Agreed.

OK, so here comes the next revision, I hope it can be put in the docs/on
the web page now:

SATA Hotplug from the User Side

In general, before unplugging a drive you must stop using it. This means
unmounting all partitions, removing the disk from a potential raid
array / device mapper / crypt setup. Also depending on your disk-bay it
might be a good idea to power down the drive using hdparm -Y followed by
a scsiadd -r.

BIG FAT WARNING: if your SATA bay does not do electrical connector
keying to let the disk firmware unload heads before the user manages to
pull it out the drive will do an emergency head unload, which is not
good and will likely reduce the drive's lifetime.

• For SIL3114 and SIL3124, AHCI and the CK804 flavour of sata_nv you -
in principle - don't have to run any commands at all. It should notice
when you yank the cable, or plug in a new device. All you have to do is
to stop using the devices before unplugging, e.g. unmount partitions on
the disk or remove the disk from a dm setup). However it is recommended
to power down the drive using hdparm -Y followed by a scsiadd -r as
stated above. One can validate which disks are attached using ``scsiadd
-p''. In the following example the disk on scsi host 3 channel 0 id 0
lun 0 will be removed:

# scsiadd -p

Attached devices:
Host: scsi2 Channel: 00 Id: 00 Lun: 00
  Vendor: ATA      Model: ST3400832AS      Rev: 3.01
  Type:   Direct-Access                    ANSI SCSI revision: 05
Host: scsi3 Channel: 00 Id: 00 Lun: 00
  Vendor: ATA      Model: ST3400620AS      Rev: 3.AA
  Type:   Direct-Access                    ANSI SCSI revision: 05

# scsiadd -r 3 0 0 0
 
Note that the device name may change from e.g. /dev/sdd to /dev/sde on a
remove/reinsert cycle (this can be fixed by using udev-provided
persistent names). Also note that it is perfectly normal to see messages
like this in dmesg:

<removing /dev/sdd>

ata4: exception Emask 0x10 SAct 0x0 SErr 0x10000 action 0x2 frozen
ata4: hard resetting port
ata4: SATA link down (SStatus 0 SControl 310)
ata4: failed to recover some devices, retrying in 5 secs
ata4: hard resetting port
ata4: SATA link down (SStatus 0 SControl 310)
ata4: failed to recover some devices, retrying in 5 secs
ata4: hard resetting port
ata4: SATA link down (SStatus 0 SControl 310)
ata4.00: disabled
ata4: EH complete
ata4.00: detaching (SCSI 3:0:0:0)
Synchronizing SCSI cache for disk sdd: 
FAILED
  status = 0, message = 00, host = 4, driver = 00
  <3>ata4: exception Emask 0x10 SAct 0x0 SErr 0x50000 action 0x2 frozen
ata4: hard resetting port
ata4: COMRESET failed (device not ready)
ata4: hardreset failed, retrying in 5 secs
ata4: hard resetting port
ata4: COMRESET failed (device not ready)
ata4: hardreset failed, retrying in 5 secs
ata4: hard resetting port

<reinserting the disk>
ata4: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
ata4.00: ATA-7, max UDMA/133, 1465149168 sectors: LBA48 NCQ (depth 0/32)
ata4.00: configured for UDMA/100
ata4: EH complete
scsi 3:0:0:0: Direct-Access     ATA      ST3750640AS      3.AA PQ: 0
ANSI: 5
SCSI device sde: 1465149168 512-byte hdwr sectors (750156 MB)
sde: Write Protect is off
sde: Mode Sense: 00 3a 00 00
SCSI device sde: drive cache: write back
SCSI device sde: 1465149168 512-byte hdwr sectors (750156 MB)
sde: Write Protect is off
sde: Mode Sense: 00 3a 00 00
SCSI device sde: drive cache: write back
 sde: unknown partition table
sd 3:0:0:0: Attached scsi disk sde
sd 3:0:0:0: Attached scsi generic sg3 type 0

However if you happen to see messages like

scsi 3:0:0:0: rejecting I/O to dead device
scsi 4:0:0:0: rejecting I/O to dead device
scsi 5:0:0:0: rejecting I/O to dead device

you did not stop using the devices before unplugging (check that you
unmounted all partitions, removed the disk from a raid array, dmsetup,
cryptsetup).  If you have no pending IO to the device, there won't be
'rejects IO to dead device' messages.

• For other chipsets one in addition might have to run scsiadd -s on
reinserting the disk.
• Note that it might not be such a good idea to do the above on drivers
which don't implement the new EH yet. Those are as of January 2007:
sata_nv, sata_promise (getting there) and sata_sx4.

Soeren (beeing stuck in a train which again seems to be stuck in a lot
of snow somewhere in southern germany...)
-- 
For the one fact about the future of which we can be certain is that it
will be utterly fantastic. -- Arthur C. Clarke, 1962

Re: SATA hotplug from the user side ?

[Henrique de Moraes Holschuh]

On Wed, 24 Jan 2007, Soeren Sonnenburg wrote:
> might be a good idea to power down the drive using hdparm -Y followed by
> a scsiadd -r.

Unfortunately, I don't think this will be the optimal way to do it for long.
Unless I am mistaken, the above will soon have the potential to spin down
the device, and then reset the device to wake it up again (through error
handling, thus causing error messages in syslog) to send it a spin down
command(!).  It will work well right now, because scsiadd -r is unable to
shutdown a device and just removes it.

In the future, you will need to do *either* one, not both, if you told the
kernel to shutdown SCSI devices (which distros will likely enable by
default, as it is the right setup for everyone but people with servers
with disks attached to multi-initiator SCSI buses and it is required for
suspend-to-disk to work).

It will be a mess to make sure everything in userspace is updated to not try
to hdparm -y (or worse, hdparm -Y) devices that the kernel will shutdown by
itself, but it will make things a lot better for the future.

Unless...

Tejun, is it feasible to teach libata to check the device power management
state before issuing any of the sleep, head unload and spin-down commands?

libata would need to block its EH from resetting a channel for this check
operation (we don't want to wake up sleeping devices to ask it if it is
sleeping...) if it cannot easily check if an device is asleep before issuing
a command to it.   Either that, or (for as long as there is no such a thing
as a multi-initiator libata device) just keep track of the device power
management state by snooping all the commands sent to it.

The idea would be to do the "optimal" thing if one tries to sleep, unload
heads or spin down an device that is already doing one of these things...
That would make things MUCH easier on userspace.

> BIG FAT WARNING: if your SATA bay does not do electrical connector
> keying to let the disk firmware unload heads before the user manages to

This either is in the SATA specification, or it is not, and according to
Tejun it probably isn't (if it were, all disks would do the right thing at
unplug).  I'd reword it to "if your SATA bay does not take extra steps to
force the disk firmware to unload heads...".

> the disk or remove the disk from a dm setup). However it is recommended
> to power down the drive using hdparm -Y followed by a scsiadd -r as
> stated above. One can validate which disks are attached using ``scsiadd

Again, this might change soon.

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh

Re: SATA hotplug from the user side ?

[Soeren Sonnenburg]

On Wed, 2007-01-24 at 13:37 -0200, Henrique de Moraes Holschuh wrote:
> On Wed, 24 Jan 2007, Soeren Sonnenburg wrote:
> > might be a good idea to power down the drive using hdparm -Y followed by
> > a scsiadd -r.
[...]
> > the disk or remove the disk from a dm setup). However it is recommended
> > to power down the drive using hdparm -Y followed by a scsiadd -r as
> > stated above. One can validate which disks are attached using ``scsiadd
> 
> Again, this might change soon.

Ok, I think someone more knowledgable than me in this area should do the
final polishing und put it in the docs.

I don't anymore see how/that I can help here.
Soeren
-- 
Sometimes, there's a moment as you're waking, when you become aware of
the real world around you, but you're still dreaming.

Re: SATA hotplug from the user side ?

[Henrique de Moraes Holschuh]

On Thu, 25 Jan 2007, Soeren Sonnenburg wrote:
> On Wed, 2007-01-24 at 13:37 -0200, Henrique de Moraes Holschuh wrote:
> > On Wed, 24 Jan 2007, Soeren Sonnenburg wrote:
> > > might be a good idea to power down the drive using hdparm -Y followed by
> > > a scsiadd -r.
> [...]
> > > the disk or remove the disk from a dm setup). However it is recommended
> > > to power down the drive using hdparm -Y followed by a scsiadd -r as
> > > stated above. One can validate which disks are attached using ``scsiadd
> > 
> > Again, this might change soon.
> 
> Ok, I think someone more knowledgable than me in this area should do the
> final polishing und put it in the docs.
> 
> I don't anymore see how/that I can help here.

You did just fine, and in fact your text is 100% correct for 2.6.20 (so it
can be taken asis for 2.6.20, and ammended later when the scsi shutdown
patch gets accepted).

So please don't fell discouraged.  The text is quite good, and if you had
not done it, it wouldn't have been improved anytime soon.

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh

Re: SATA hotplug from the user side ?

[Eric D. Mudama]

On 1/24/07, Henrique de Moraes Holschuh <hmh@hmh.eng.br> wrote:
> Tejun, is it feasible to teach libata to check the device power management
> state before issuing any of the sleep, head unload and spin-down commands?
>
> libata would need to block its EH from resetting a channel for this check
> operation (we don't want to wake up sleeping devices to ask it if it is
> sleeping...) if it cannot easily check if an device is asleep before issuing
> a command to it.   Either that, or (for as long as there is no such a thing
> as a multi-initiator libata device) just keep track of the device power
> management state by snooping all the commands sent to it.

In the ATA spec, the CHECK POWER MODE command (E5h) specifically
states that it shouldn't alter the power mode of the device.
Therefore, there should be no harm in checking the power mode of a
device in STANDBY or IDLE state, and it shouldn't cause the device to
spin up.  Checking a drive on my bench I confirmed this behavior.

With Serial ATA you have to consider the SATA bus PM states as well,
but that's a different topic of discussion.

--eric

Re: SATA hotplug from the user side ?

[Tejun Heo]

Eric D. Mudama wrote:
> In the ATA spec, the CHECK POWER MODE command (E5h) specifically
> states that it shouldn't alter the power mode of the device.
> Therefore, there should be no harm in checking the power mode of a
> device in STANDBY or IDLE state, and it shouldn't cause the device to
> spin up.  Checking a drive on my bench I confirmed this behavior.

Yes, we can but...

1. As I wrote in the other reply, I don't think that's necessary.
Devices don't spin up on FLUSH_CACHE followed by STANDBY_IMMEDIATE when
they are in Standby state. (Please point out if this isn't true on most
drives)

2. As the code currently stands, we have to rely on SCSI HLD for
shutting down.  We can query power state there and issue shutdown
conditionally and all those will be properly translated by libata but
it's a bit complex.  I'm not sure whether the added complexity is justified.

> With Serial ATA you have to consider the SATA bus PM states as well,
> but that's a different topic of discussion.

I think that can be dealt with separately.

Thanks.

-- 
tejun

Re: SATA hotplug from the user side ?

[Tejun Heo]

Hello, Henrique, sorry about being late.

Henrique de Moraes Holschuh wrote:
> Tejun, is it feasible to teach libata to check the device power management
> state before issuing any of the sleep, head unload and spin-down commands?
> 
> libata would need to block its EH from resetting a channel for this check
> operation (we don't want to wake up sleeping devices to ask it if it is
> sleeping...) if it cannot easily check if an device is asleep before issuing
> a command to it.   Either that, or (for as long as there is no such a thing
> as a multi-initiator libata device) just keep track of the device power
> management state by snooping all the commands sent to it.
> 
> The idea would be to do the "optimal" thing if one tries to sleep, unload
> heads or spin down an device that is already doing one of these things...
> That would make things MUCH easier on userspace.

For that to be meaningful, the premise is that ATA drives spin up to
process STANDBY IMMEDIATE when it's in Standby state.  The spec doesn't
exactly specify what happens on when STANDBY IMMEDIATE is received while
in Standby state other than it exits to Active iff Media access is required.

I've tested several drivers from different vendors and none of them
seems to spin up when they receive FLUSH and STANDBY IMMEDIATE while
they're in Standby mode.  They just ack that the command is complete and
stay spun down.  Do you have drives which act differently?

So, if most of drives act that way, whether both or anyone of them are
being done just doesn't matter as long as at least one gets done.

Thanks.

-- 
tejun

Re: SATA hotplug from the user side ?

[Tejun Heo]

Soeren Sonnenburg wrote:
> OK, how about this (please especially check the non SIL part):
> 
> SATA Hotplug from the User Side
> 
> - For SIL3114 and SIL3124 you don't have to run any commands at all. It

ahci and ck804 flavor of sata_nv's can do hotplug without user
assistance too.

[--snip--]
> - For other chipsets one in addition to stop using the device before
> unplugging has to call scsiadd -r to fully remove it, e.g. in the
> following example the disk on scsi host 3 channel 0 id 0 lun 0 will be
> removed, then on reinserting a disk call scsiadd -s :
> 
> # scsiadd -p
> 
> Attached devices:
> Host: scsi2 Channel: 00 Id: 00 Lun: 00
>   Vendor: ATA      Model: ST3400832AS      Rev: 3.01
>   Type:   Direct-Access                    ANSI SCSI revision: 05
> Host: scsi3 Channel: 00 Id: 00 Lun: 00
>   Vendor: ATA      Model: ST3400620AS      Rev: 3.AA
>   Type:   Direct-Access                    ANSI SCSI revision: 05
> 
> # scsiadd -r 3 0 0 0
> # scsiadd -s

Doing the above might not be such a good idea on drivers which don't
implement new EH yet.  Those are sata_mv, sata_promise (getting there)
and sata_sx4.

Thanks.

-- 
tejun

Re: SATA hotplug from the user side ?

[Robert Hancock]

Soeren Sonnenburg wrote:
> • Note that it might not be such a good idea to do the above on drivers
> which don't implement the new EH yet. Those are as of January 2007:
> sata_nv, sata_promise (getting there) and sata_sx4.

That should be sata_mv not sata_nv.

-- 
Robert Hancock      Saskatoon, SK, Canada
To email, remove "nospam" from hancockr@nospamshaw.ca
Home Page: http://www.roberthancock.com/