blacklist of hard drives that misreport serial numbers

blacklist of hard drives that misreport serial numbers

[Ethan Baldridge]

Does such a list exist?

I've got a system that depends on the symlink in /dev/disk/by-id to
identify which of the external drives at our datacenter contains the
data that needs to be removed and shipped out after we export to the
drive.

This worked fabulously during in-house testing, but now that the system
is in production we find that some Toshiba drives (at least) don't
report a serial number to the OS that looks anything like what is
printed on the outside of the drive itself. That complicates things.

Is this dastardly behavior limited to Toshiba, or merely the specific
model in question, or is it a known widespread problem for drives from
any vendor (and is it at least guaranteed that certain models are safe)?

Thanks,
Ethan Baldridge

Re: blacklist of hard drives that misreport serial numbers

[David Zeuthen]

Hi,

On Thu, Jun 10, 2010 at 10:58 PM, Ethan Baldridge
<baldridge.ethan@gmail.com> wrote:
> Does such a list exist?
>
> I've got a system that depends on the symlink in /dev/disk/by-id to
> identify which of the external drives at our datacenter contains the
> data that needs to be removed and shipped out after we export to the
> drive.
>
> This worked fabulously during in-house testing, but now that the system
> is in production we find that some Toshiba drives (at least) don't
> report a serial number to the OS that looks anything like what is
> printed on the outside of the drive itself. That complicates things.
>
> Is this dastardly behavior limited to Toshiba, or merely the specific
> model in question, or is it a known widespread problem for drives from
> any vendor (and is it at least guaranteed that certain models are safe)?

For SCSI disks (e.g. data obtained via scsi_id), ID_SERIAL isn't
really the page 0x80 serial number that is typically printed on the
label of the hard disk. This commit has some more information

http://git.kernel.org/?p=linux/hotplug/udev.git;a=commitdiff;hN9fdfccbdd16f0cfdb5c8fa8484a8ba0f2e69d3

and exports the serial number and WWN in ID_WWN and ID_SCSI_SERIAL.

FWIW, I don't think we use ID_SCSI_SERIAL in any /dev/disk symlinks
but we probably could. Any chance you could try seeing if
ID_SCSI_SERIAL works for you? It's available since udev version 148 or
later.

Also, we do use ID_WWN for /dev/disk/by-id/wwn-0x* symlinks (and this
is set for both SCSI (including e.g. SAS) through scsi_id and ATA
through ata_id) - maybe you can use the by-id/wwn-* symlinks instead?

    David

Re: blacklist of hard drives that misreport serial numbers

[Ethan Baldridge]

On Thu, 2010-06-10 at 23:36 -0400, David Zeuthen wrote:
> Hi,
> 
> On Thu, Jun 10, 2010 at 10:58 PM, Ethan Baldridge
> <baldridge.ethan@gmail.com> wrote:
> > Does such a list exist?
> >
> > I've got a system that depends on the symlink in /dev/disk/by-id to
> > identify which of the external drives at our datacenter contains the
> > data that needs to be removed and shipped out after we export to the
> > drive.
> >
> > This worked fabulously during in-house testing, but now that the system
> > is in production we find that some Toshiba drives (at least) don't
> > report a serial number to the OS that looks anything like what is
> > printed on the outside of the drive itself. That complicates things.
> >
> > Is this dastardly behavior limited to Toshiba, or merely the specific
> > model in question, or is it a known widespread problem for drives from
> > any vendor (and is it at least guaranteed that certain models are safe)?
> 
> For SCSI disks (e.g. data obtained via scsi_id), ID_SERIAL isn't
> really the page 0x80 serial number that is typically printed on the
> label of the hard disk. This commit has some more information
> 
> http://git.kernel.org/?p=linux/hotplug/udev.git;a=commitdiff;hN9fdfccbdd16f0cfdb5c8fa8484a8ba0f2e69d3
> 
> and exports the serial number and WWN in ID_WWN and ID_SCSI_SERIAL.
> 
> FWIW, I don't think we use ID_SCSI_SERIAL in any /dev/disk symlinks
> but we probably could. Any chance you could try seeing if
> ID_SCSI_SERIAL works for you? It's available since udev version 148 or
> later.
> 
> Also, we do use ID_WWN for /dev/disk/by-id/wwn-0x* symlinks (and this
> is set for both SCSI (including e.g. SAS) through scsi_id and ATA
> through ata_id) - maybe you can use the by-id/wwn-* symlinks instead?
> 
>     David

Thanks for the info, David, but unfortunately these are USB drives, and
scsi_id doesn't seem to return anything (although this is with udev
version 147, so I can try upgrading, but that patch only seems to affect
the scsi subsystem - has something similar been done to usb?)

$ /lib/udev/usb_id -x /block/sdg
ID_VENDOR=Toshiba
ID_VENDOR_ENC=Toshiba\x20
ID_VENDOR_ID\x0480
ID_MODEL=External_USB_HDD
ID_MODEL_ENC=External\x20USB\x20HDD
ID_MODEL_ID 03
ID_REVISION\x010f
ID_SERIAL=Toshiba_External_USB_HDD_2010040104E4-0:0
ID_SERIAL_SHORT 10040104E4
ID_TYPE=disk
ID_INSTANCE=0:0
ID_BUS=usb
ID_USB_INTERFACES=:080650:
ID_USB_INTERFACE_NUM\0
ID_USB_DRIVER=usb-storage

$ /lib/udev/scsi_id -x /block/sdg
$

Versus this result for a Seagate drive:
$ /lib/udev/usb_id -x /block/sdf
ID_VENDOR=Seagate
ID_VENDOR_ENC=Seagate\x20
ID_VENDOR_ID\vc2
ID_MODEL=FreeAgentDesktop
ID_MODEL_ENC=FreeAgentDesktop
ID_MODEL_ID000
ID_REVISION\x100F
ID_SERIAL=Seagate_FreeAgentDesktop_5QM0A7X7-0:0
ID_SERIAL_SHORT=5QM0A7X7
ID_TYPE=disk
ID_INSTANCE=0:0
ID_BUS=usb
ID_USB_INTERFACES=:080650:
ID_USB_INTERFACE_NUM\0
ID_USB_DRIVER=usb-storage

And this number actually does match what one would expect by looking at
the bottom of the drive enclosure.

The serial is used in the symlinks in by-id here:
lrwxrwxrwx 1 root root  9 2010-05-13 14:32
usb-Seagate_FreeAgentDesktop_5QM0A7X7-0:0 -> ../../sdf
lrwxrwxrwx 1 root root 10 2010-05-13 14:32
usb-Seagate_FreeAgentDesktop_5QM0A7X7-0:0-part1 -> ../../sdf1
lrwxrwxrwx 1 root root  9 2010-05-20 17:25
usb-Toshiba_External_USB_HDD_2010040104E4-0:0 -> ../../sdg
lrwxrwxrwx 1 root root 10 2010-05-20 17:25
usb-Toshiba_External_USB_HDD_2010040104E4-0:0-part1 -> ../../sdg1
lrwxrwxrwx 1 root root  9 2010-06-09 10:37
usb-Toshiba_External_USB_HDD_2010040105DD-0:0 -> ../../sde
lrwxrwxrwx 1 root root 10 2010-06-09 10:37
usb-Toshiba_External_USB_HDD_2010040105DD-0:0-part1 -> ../../sde1


Thanks,
Ethan

Re: blacklist of hard drives that misreport serial numbers

[David Zeuthen]

Hey,

On Thu, Jun 10, 2010 at 11:57 PM, Ethan Baldridge
<baldridge.ethan@gmail.com> wrote:
> Thanks for the info, David, but unfortunately these are USB drives, and
> scsi_id doesn't seem to return anything (although this is with udev
> version 147, so I can try upgrading, but that patch only seems to affect
> the scsi subsystem - has something similar been done to usb?)

I think that we always use the USB serial number for any device
connected via USB. I remember something about that if the USB device
is a ATA disk and the bridge is sufficiently compatible, it should be
possible to retrieve the ATA serial number from IDENTIFY data.
Presumably it's a simple matter of attempting to run ata_id (which I
don't think we currently do).

Try checking the devices with ata_id (from udev), hdparm(8) and
skdump(1) and check if the serial reported by them is what you're
after?

(I'm mentioning skdump(1) because it has some extra secret sauce for
dealing with ATA devices behind certain kinds of USB bridges.)

> And this number actually does match what one would expect by looking at
> the bottom of the drive enclosure.

This is probably just a coincidence. In general I don't think you can
make any assumptions about what kind of serial number vendors put on
the *enclosure*. If it was ATA or SAS disks you could probably expect
the vendors to put the serial number or WWN on the hard disks itself.

That said, in the case where we can extract both the ATA serial number
and the USB serial number for a disk, I think we should prefer the ATA
one. Or maybe, for symlinks at least, we should use both. Kay, what do
you think?

     David

Re: blacklist of hard drives that misreport serial numbers

[Ethan Baldridge]

Thanks, David,

On Fri, 2010-06-11 at 18:38 -0400, David Zeuthen wrote:
> Hey,
> 
> On Thu, Jun 10, 2010 at 11:57 PM, Ethan Baldridge
> <baldridge.ethan@gmail.com> wrote:
> > Thanks for the info, David, but unfortunately these are USB drives, and
> > scsi_id doesn't seem to return anything (although this is with udev
> > version 147, so I can try upgrading, but that patch only seems to affect
> > the scsi subsystem - has something similar been done to usb?)
> 
> I think that we always use the USB serial number for any device
> connected via USB. I remember something about that if the USB device
> is a ATA disk and the bridge is sufficiently compatible, it should be
> possible to retrieve the ATA serial number from IDENTIFY data.
> Presumably it's a simple matter of attempting to run ata_id (which I
> don't think we currently do).
> 
> Try checking the devices with ata_id (from udev), hdparm(8) and
> skdump(1) and check if the serial reported by them is what you're
> after?
> 

ata_id returns nothing for any of the USB drives.
hdparm also gives nothing for any of them except:
 HDIO_DRIVE_CMD(identify) failed: Invalid exchange
 HDIO_GET_IDENTITY failed: Invalid argument

skdump gave much more interesting results.

For the Seagate drive (which is already returning the "proper" - or at
least expected - serial number) it gave a whole battery of SMART data,
as well as Model: [ST3500820AS], and Serial: [5QM0A7X7]

For the Toshiba drives (which do not have the expected serial number) I
only got:
$ sudo skdump /dev/sdg
Device: sat12:/dev/sdg
Type: 12 Byte SCSI ATA SAT Passthru
Size: 305245 MiB
Awake: Operation not supported
ATA SMART not supported.

So it seems like that's also a dead-end. I think these drives just don't
give useful information to the operating system. Of course, you can
disagree and say that in reality they don't have useful information
printed on them, but either way it comes down to the same thing as far
as I can see - "don't buy these if you care what the serial number is."

The drives are the pocket-size ones in a solid molded enclosure with no
way to get at the actual drive, so it's definitely not helpful to know
that *if* I could take it apart and look at the drive, it *might* have
the number I want printed on it. Toshiba should really put that on the
outside (as Seagate apparently does, at least for one model.)

Thanks,
Ethan